Jul 28 17:08:23.076462 ip-10-0-135-73 systemd[1]: Starting Kubernetes Kubelet... Jul 28 17:08:23.470820 ip-10-0-135-73 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:08:23.470820 ip-10-0-135-73 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 17:08:23.470820 ip-10-0-135-73 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:08:23.470820 ip-10-0-135-73 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 17:08:23.470820 ip-10-0-135-73 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:08:23.472941 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.472832 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480261 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480288 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480293 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480296 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480299 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480302 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480306 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:23.480295 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480310 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480313 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480316 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480318 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480321 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480323 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480352 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480357 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480361 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480364 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480367 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480370 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480373 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480376 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480379 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480382 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480385 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480388 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:23.480607 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480392 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480396 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480399 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480402 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480405 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480408 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480411 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480413 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480416 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480420 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480422 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480425 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480427 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480430 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480433 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480437 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480440 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480442 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480446 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480448 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480451 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:23.481049 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480454 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480457 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480459 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480462 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480465 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480468 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480471 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480473 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480476 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480478 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480481 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480483 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480486 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480489 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480492 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480495 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480497 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480500 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480502 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480505 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:23.481610 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480507 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480510 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480513 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480517 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480520 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480523 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480526 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480528 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480531 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480533 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480536 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480539 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480542 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480544 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480547 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480549 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480552 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480556 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480558 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480561 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:23.482114 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480563 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480566 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480570 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480573 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480576 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480578 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480581 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480583 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480587 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480591 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480594 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480597 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.480600 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482265 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482275 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482278 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482281 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482284 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482287 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:23.482657 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482290 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482292 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482295 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482297 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482300 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482304 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482308 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482311 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482314 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482317 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482320 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482324 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482327 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482329 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482332 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482335 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482338 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482341 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482344 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:23.483123 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482346 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482349 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482352 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482355 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482357 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482360 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482362 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482365 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482367 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482371 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482374 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482377 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482379 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482382 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482384 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482387 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482389 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482392 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482395 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482397 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:23.483623 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482399 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482402 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482404 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482407 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482409 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482413 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482415 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482418 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482421 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482424 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482426 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482429 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482432 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482435 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482438 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482440 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482443 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482445 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482449 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482452 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:23.484144 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482454 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482457 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482459 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482463 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482467 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482470 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482472 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482475 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482479 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482482 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482485 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482489 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482491 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482494 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482496 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482499 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482502 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482504 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482508 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:23.484638 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482510 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482514 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482516 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482519 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482522 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482524 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482527 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482529 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482532 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482534 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482537 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482539 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482542 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482545 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482547 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482958 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482963 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482966 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482969 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482971 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:23.485129 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482974 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482977 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482979 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482982 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482985 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482987 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482991 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482995 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.482998 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483000 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483003 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483006 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483010 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483013 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483015 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483018 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483021 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483024 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483027 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483029 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:23.485637 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483032 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483035 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483038 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483040 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483043 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483046 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483048 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483053 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483057 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483060 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483063 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483065 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483068 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483070 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483073 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483075 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483078 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483081 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:23.486176 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483084 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483087 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483090 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483093 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483096 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483099 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483102 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483122 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483126 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483129 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483132 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483135 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483138 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483140 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483143 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483146 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483148 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483151 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483154 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483158 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:23.486628 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483160 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483163 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483166 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483168 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483171 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483173 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483176 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483179 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483181 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483184 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483187 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483189 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483192 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483195 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483197 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483200 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483202 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483205 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483208 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483212 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:23.487136 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483215 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483217 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483220 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483222 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483225 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483227 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483230 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483232 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483235 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483237 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483240 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483242 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483245 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483247 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483250 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483252 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483331 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483339 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483347 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483352 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483356 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 17:08:23.487660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483359 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483363 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483368 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483371 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483374 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483377 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483381 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483384 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483389 2579 flags.go:64] FLAG: --cgroup-root="" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483393 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483396 2579 flags.go:64] FLAG: --client-ca-file="" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483400 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483403 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483408 2579 flags.go:64] FLAG: --cluster-domain="" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483411 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483415 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483418 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483421 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483425 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483429 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483432 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483435 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483438 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483441 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 17:08:23.488197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483445 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483448 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483451 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483455 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483458 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483461 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483464 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483467 2579 flags.go:64] FLAG: --enable-server="true" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483470 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483475 2579 flags.go:64] FLAG: --event-burst="100" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483478 2579 flags.go:64] FLAG: --event-qps="50" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483481 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483484 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483488 2579 flags.go:64] FLAG: --eviction-hard="" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483491 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483495 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483501 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483504 2579 flags.go:64] FLAG: --eviction-soft="" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483508 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483511 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483514 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483517 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483520 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483523 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483526 2579 flags.go:64] FLAG: --feature-gates="" Jul 28 17:08:23.488821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483530 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483533 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483536 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483539 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483543 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483546 2579 flags.go:64] FLAG: --help="false" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483549 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483552 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483555 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483557 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483561 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483565 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483568 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483571 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483574 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483576 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483579 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483583 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483586 2579 flags.go:64] FLAG: --kube-reserved="" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483589 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483592 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483595 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483597 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483602 2579 flags.go:64] FLAG: --lock-file="" Jul 28 17:08:23.489480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483605 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483608 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483611 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483617 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483621 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483624 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483627 2579 flags.go:64] FLAG: --logging-format="text" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483631 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483634 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483637 2579 flags.go:64] FLAG: --manifest-url="" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483640 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483645 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483648 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483653 2579 flags.go:64] FLAG: --max-pods="110" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483656 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483659 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483662 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483664 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483668 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483670 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483673 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483682 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483685 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483688 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 17:08:23.490084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483691 2579 flags.go:64] FLAG: --pod-cidr="" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483694 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483701 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483705 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483709 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483712 2579 flags.go:64] FLAG: --port="10250" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483715 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483719 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a6e1c098684f856" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483722 2579 flags.go:64] FLAG: --qos-reserved="" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483725 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483728 2579 flags.go:64] FLAG: --register-node="true" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483731 2579 flags.go:64] FLAG: --register-with-taints="" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483735 2579 flags.go:64] FLAG: --registry-burst="10" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483739 2579 flags.go:64] FLAG: --registry-qps="5" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483742 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483745 2579 flags.go:64] FLAG: --reserved-memory="" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483748 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483751 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483754 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483757 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483760 2579 flags.go:64] FLAG: --runonce="false" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483763 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483766 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483769 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483772 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 17:08:23.490721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483775 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483778 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483781 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483784 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483787 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483790 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483793 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483796 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483799 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483802 2579 flags.go:64] FLAG: --system-cgroups="" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483806 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483812 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483815 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483819 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483824 2579 flags.go:64] FLAG: --tls-min-version="" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483827 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483830 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483833 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483836 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483839 2579 flags.go:64] FLAG: --v="2" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483844 2579 flags.go:64] FLAG: --version="false" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483856 2579 flags.go:64] FLAG: --vmodule="" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483865 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.483868 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483990 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:23.491385 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483994 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.483999 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484003 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484007 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484009 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484012 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484015 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484018 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484022 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484025 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484028 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484031 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484034 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484037 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484040 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484043 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484046 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484051 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484053 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:23.492037 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484056 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484059 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484063 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484066 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484068 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484071 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484074 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484077 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484079 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484082 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484085 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484088 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484090 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484093 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484096 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484098 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484101 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484116 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484119 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484122 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:23.492519 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484125 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484127 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484130 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484134 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484138 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484141 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484143 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484146 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484150 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484152 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484156 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484159 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484162 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484165 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484169 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484172 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484175 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484177 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484180 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:23.493073 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484183 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484186 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484188 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484191 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484195 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484198 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484201 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484203 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484206 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484208 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484211 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484213 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484216 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484219 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484222 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484224 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484227 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484229 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484232 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484235 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:23.493555 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484237 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484240 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484243 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484248 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484250 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484253 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484255 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484259 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484262 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484265 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484267 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484270 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484272 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484275 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484278 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484280 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484282 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484287 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484290 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:23.494057 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.484292 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.484302 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.491625 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.491646 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.492460 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.492471 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492537 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492544 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492548 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492551 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492554 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492557 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492561 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492564 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492567 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:23.494559 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492570 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492573 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492576 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492579 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492582 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492585 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492588 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492591 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492593 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492596 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492599 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492601 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492604 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492607 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492609 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492612 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492615 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492617 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492620 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492623 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:23.494949 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492627 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492630 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492633 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492636 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492639 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492642 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492644 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492647 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492651 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492655 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492659 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492663 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492666 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492669 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492672 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492675 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492678 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492681 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492684 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:23.495462 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492686 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492689 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492691 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492694 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492697 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492700 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492704 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492708 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492710 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492713 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492716 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492718 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492721 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492724 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492728 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492731 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492734 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492737 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492740 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:23.495932 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492743 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492746 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492749 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492751 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492754 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492757 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492760 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492763 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492766 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492769 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492771 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492774 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492776 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492779 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492782 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492784 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492787 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492789 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492792 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492795 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:23.496430 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492797 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492800 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492802 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492805 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492807 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492810 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492813 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492816 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492819 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492822 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492824 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492827 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.492832 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492957 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:08:23.496973 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492963 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492968 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492972 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492976 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492979 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492982 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492985 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492987 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492990 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492992 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492995 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.492997 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493000 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493002 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493005 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493008 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493010 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493013 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493015 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:08:23.497335 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493018 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493020 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493023 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493025 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493029 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493031 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493035 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493037 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493040 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493043 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493046 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493049 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493051 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493054 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493057 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493059 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493062 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493064 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493067 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:08:23.497804 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493069 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493072 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493074 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493077 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493080 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493082 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493085 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493088 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493090 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493095 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493098 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493101 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493119 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493124 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493127 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493130 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493133 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493136 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493139 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493141 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:08:23.498303 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493144 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493147 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493151 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493154 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493157 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493159 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493162 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493165 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493167 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493169 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493172 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493175 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493177 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493180 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493182 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493185 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493187 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493190 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493193 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493195 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:08:23.498797 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493198 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493201 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493203 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493206 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493209 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493212 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493214 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493217 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493219 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493222 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493225 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493228 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493230 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493233 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493236 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493239 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493242 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493245 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493247 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:08:23.499301 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:23.493250 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:08:23.499776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.493255 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:08:23.499776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.493391 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 17:08:23.499776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.495413 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 17:08:23.499776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.496767 2579 server.go:1019] "Starting client certificate rotation" Jul 28 17:08:23.499776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.496864 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:08:23.499776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.496908 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:08:23.499776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.499343 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:08:23.502393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.502375 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:08:23.513964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.513937 2579 log.go:25] "Validated CRI v1 runtime API" Jul 28 17:08:23.517264 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.517239 2579 log.go:25] "Validated CRI v1 image API" Jul 28 17:08:23.518245 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.518222 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 17:08:23.521293 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.521272 2579 fs.go:135] Filesystem UUIDs: map[44e46f9e-8976-4815-adf8-dba6fdf369ba:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d008c504-da3c-4374-8392-5953492193e1:/dev/nvme0n1p4] Jul 28 17:08:23.521363 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.521292 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 17:08:23.526627 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.526484 2579 manager.go:216] Machine: {Timestamp:2026-07-28 17:08:23.525149239 +0000 UTC m=+0.328837820 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098865 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c6c89165584773bdf9374a85f7d5 SystemUUID:ec23c6c8-9165-5847-73bd-f9374a85f7d5 BootID:ed8d73d0-7916-4e7b-90e8-0f1658bc4cc5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:84:de:8f:e1:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:84:de:8f:e1:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:65:5a:e9:9f:1b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 17:08:23.526627 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.526611 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 17:08:23.526825 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.526712 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 17:08:23.529644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.529601 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 17:08:23.529848 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.529648 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-73.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 17:08:23.530491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.530479 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 17:08:23.530527 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.530493 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 17:08:23.530527 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.530500 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:08:23.531134 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.531121 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:08:23.532337 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.532319 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:08:23.532489 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.532476 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 17:08:23.534388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.534373 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:08:23.534521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.534511 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 17:08:23.534556 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.534544 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 28 17:08:23.534556 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.534554 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 17:08:23.534608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.534570 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 17:08:23.534608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.534579 2579 kubelet.go:398] "Adding apiserver pod source" Jul 28 17:08:23.534608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.534601 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 17:08:23.535034 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.535010 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:08:23.535034 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.535040 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:08:23.537519 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.537501 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 17:08:23.539545 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.539528 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 17:08:23.539621 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.539560 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 17:08:23.541014 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541003 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 17:08:23.541051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541018 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 17:08:23.541051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541025 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 17:08:23.541051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541031 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 17:08:23.541051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541041 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 17:08:23.541051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541046 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 17:08:23.541200 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541055 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 17:08:23.541200 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541060 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 17:08:23.541200 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541066 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 17:08:23.541200 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541071 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 17:08:23.541200 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541096 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 17:08:23.541200 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541115 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 17:08:23.541758 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541749 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 17:08:23.541758 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.541758 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 17:08:23.545099 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.545078 2579 server.go:1268] "Started kubelet" Jul 28 17:08:23.545358 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.545327 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 17:08:23.545557 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.545541 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 17:08:23.545702 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.545667 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 17:08:23.545769 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.545457 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 17:08:23.545769 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.545569 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-73.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 17:08:23.545831 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.545787 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 17:08:23.545971 ip-10-0-135-73 systemd[1]: Started Kubernetes Kubelet. Jul 28 17:08:23.549882 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.549853 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 17:08:23.551838 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.551816 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 28 17:08:23.552515 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.552492 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 17:08:23.557588 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.557566 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 17:08:23.557717 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.557640 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 17:08:23.557717 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.557715 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:23.558296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.558276 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 17:08:23.558945 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.558790 2579 factory.go:55] Registering systemd factory Jul 28 17:08:23.558945 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.558806 2579 factory.go:223] Registration of the systemd container factory successfully Jul 28 17:08:23.558945 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.558863 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 17:08:23.558945 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.558873 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 17:08:23.559140 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559018 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 28 17:08:23.559140 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559025 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 28 17:08:23.559237 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559182 2579 factory.go:153] Registering CRI-O factory Jul 28 17:08:23.559237 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559199 2579 factory.go:223] Registration of the crio container factory successfully Jul 28 17:08:23.559237 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.557374 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-73.ec2.internal.18c68326780c2614 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-73.ec2.internal,UID:ip-10-0-135-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-73.ec2.internal,},FirstTimestamp:2026-07-28 17:08:23.545038356 +0000 UTC m=+0.348726953,LastTimestamp:2026-07-28 17:08:23.545038356 +0000 UTC m=+0.348726953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-73.ec2.internal,}" Jul 28 17:08:23.559394 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559250 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 17:08:23.559394 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559274 2579 factory.go:103] Registering Raw factory Jul 28 17:08:23.559394 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559290 2579 manager.go:1196] Started watching for new ooms in manager Jul 28 17:08:23.559701 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.559681 2579 manager.go:349] Starting recovery of all containers Jul 28 17:08:23.561954 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.561925 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 17:08:23.562266 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.562215 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-73.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 17:08:23.568812 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.568748 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmrk8" Jul 28 17:08:23.569453 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.569433 2579 manager.go:354] Recovery completed Jul 28 17:08:23.574337 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.574315 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:23.576832 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.576809 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:23.576946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.576850 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:23.576946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.576864 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:23.577546 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.577527 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 17:08:23.577546 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.577544 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:23.577714 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.577543 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 17:08:23.577714 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.577622 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:08:23.578407 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.578369 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:23.578488 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.578430 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:23.578488 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.578443 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:23.579563 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.579549 2579 policy_none.go:49] "None policy: Start" Jul 28 17:08:23.579643 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.579567 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 17:08:23.579643 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.579581 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 17:08:23.579832 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.579813 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mmrk8" Jul 28 17:08:23.579985 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.579910 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-73.ec2.internal.18c6832679f14764 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-73.ec2.internal,UID:ip-10-0-135-73.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-73.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-73.ec2.internal,},FirstTimestamp:2026-07-28 17:08:23.576831844 +0000 UTC m=+0.380520432,LastTimestamp:2026-07-28 17:08:23.576831844 +0000 UTC m=+0.380520432,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-73.ec2.internal,}" Jul 28 17:08:23.581214 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.581202 2579 policy_none.go:47] "Start" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.616536 2579 manager.go:337] "Starting Device Plugin manager" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.616570 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.616583 2579 server.go:87] "Starting device plugin registration server" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.616894 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.616907 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.616993 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.617084 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.617095 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.617623 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 17:08:23.628329 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.617675 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:23.676979 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.676941 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 17:08:23.678197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.678173 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 17:08:23.678336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.678214 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 17:08:23.678336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.678239 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 17:08:23.678336 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.678286 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 17:08:23.682683 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.682664 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:23.717935 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.717907 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:23.718925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.718905 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:23.719035 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.718947 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:23.719035 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.718962 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:23.719035 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.718991 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.729690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.729604 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.729690 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.729636 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-73.ec2.internal\": node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:23.748700 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.748673 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:23.779273 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.779234 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal"] Jul 28 17:08:23.779435 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.779313 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:23.780948 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.780927 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:23.780948 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.780958 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:23.781099 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.780968 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:23.782291 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.782276 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:23.782397 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.782382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.782441 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.782418 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:23.783161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.783141 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:23.783161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.783170 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:23.783309 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.783180 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:23.783309 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.783146 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:23.783309 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.783242 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:23.783309 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.783252 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:23.784438 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.784423 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.784480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.784456 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:08:23.785266 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.785247 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:08:23.785370 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.785279 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:08:23.785370 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.785293 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:08:23.808120 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.808080 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-73.ec2.internal\" not found" node="ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.812686 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.812665 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-73.ec2.internal\" not found" node="ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.848936 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.848903 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:23.860319 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.860287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fcf926d021c29c5f806efa917b417f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal\" (UID: \"4fcf926d021c29c5f806efa917b417f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.860402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.860329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df396ec83974a4b79253cc8ed0dbbedd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-73.ec2.internal\" (UID: \"df396ec83974a4b79253cc8ed0dbbedd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.860402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.860345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4fcf926d021c29c5f806efa917b417f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal\" (UID: \"4fcf926d021c29c5f806efa917b417f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.949573 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:23.949539 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:23.960957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.960927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df396ec83974a4b79253cc8ed0dbbedd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-73.ec2.internal\" (UID: \"df396ec83974a4b79253cc8ed0dbbedd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.961043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.960942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/df396ec83974a4b79253cc8ed0dbbedd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-73.ec2.internal\" (UID: \"df396ec83974a4b79253cc8ed0dbbedd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.961043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.960988 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4fcf926d021c29c5f806efa917b417f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal\" (UID: \"4fcf926d021c29c5f806efa917b417f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.961043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.961007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fcf926d021c29c5f806efa917b417f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal\" (UID: \"4fcf926d021c29c5f806efa917b417f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.961169 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.961048 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fcf926d021c29c5f806efa917b417f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal\" (UID: \"4fcf926d021c29c5f806efa917b417f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:23.961169 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:23.961065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4fcf926d021c29c5f806efa917b417f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal\" (UID: \"4fcf926d021c29c5f806efa917b417f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:24.050712 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.050622 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.111175 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.111132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:24.116180 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.116161 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" Jul 28 17:08:24.151304 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.151264 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.251870 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.251822 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.352254 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.352211 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.452577 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.452554 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.496913 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.496884 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 17:08:24.497344 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.497047 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:08:24.553192 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.553158 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.564614 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.564583 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:24.584158 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.584098 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:03:23 +0000 UTC" deadline="2028-01-30 05:48:42.332241062 +0000 UTC" Jul 28 17:08:24.584158 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.584152 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13212h40m17.748093526s" Jul 28 17:08:24.653419 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.653342 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.738175 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.738145 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:24.753694 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.753657 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.757392 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:24.757365 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcf926d021c29c5f806efa917b417f7.slice/crio-c58f5273c3f61cf637e6ed02bd6259d4c2573f04d6cc9ef5cff4bfd43f786868 WatchSource:0}: Error finding container c58f5273c3f61cf637e6ed02bd6259d4c2573f04d6cc9ef5cff4bfd43f786868: Status 404 returned error can't find the container with id c58f5273c3f61cf637e6ed02bd6259d4c2573f04d6cc9ef5cff4bfd43f786868 Jul 28 17:08:24.762543 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.762529 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:08:24.854709 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:24.854675 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-73.ec2.internal\" not found" Jul 28 17:08:24.919973 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.919889 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:24.954926 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:24.954894 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf396ec83974a4b79253cc8ed0dbbedd.slice/crio-82dc6ea9f1dac21c5dfeff7e15b03d692cc90342aa9630eb215b742a45b67b37 WatchSource:0}: Error finding container 82dc6ea9f1dac21c5dfeff7e15b03d692cc90342aa9630eb215b742a45b67b37: Status 404 returned error can't find the container with id 82dc6ea9f1dac21c5dfeff7e15b03d692cc90342aa9630eb215b742a45b67b37 Jul 28 17:08:24.958090 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.958066 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" Jul 28 17:08:24.974672 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.974645 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:08:24.975777 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.975763 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" Jul 28 17:08:24.997174 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:24.997148 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:08:25.310992 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.310912 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:08:25.536153 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.536121 2579 apiserver.go:52] "Watching apiserver" Jul 28 17:08:25.550042 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.550009 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 17:08:25.551849 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.551813 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-b585j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6","openshift-image-registry/node-ca-6clsf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal","openshift-multus/multus-9ntls","openshift-multus/network-metrics-daemon-g2b76","openshift-network-diagnostics/network-check-target-2mnrj","kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal","openshift-cluster-node-tuning-operator/tuned-hgq9x","openshift-dns/node-resolver-v9vdk","openshift-multus/multus-additional-cni-plugins-29j4x","openshift-network-operator/iptables-alerter-gvh5c","openshift-ovn-kubernetes/ovnkube-node-lbjhs","kube-system/global-pull-secret-syncer-djz6z"] Jul 28 17:08:25.553521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.553496 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.555024 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.554995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.556417 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.556119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.557340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.557320 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.557444 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.557409 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:25.557515 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.557460 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n6fwb\"" Jul 28 17:08:25.557574 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.557535 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 17:08:25.558187 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.558158 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 17:08:25.558647 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.558629 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 17:08:25.558950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.558930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.560920 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.560898 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 17:08:25.560920 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.560913 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 17:08:25.561102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.560980 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 17:08:25.561190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.561134 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 17:08:25.561190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.561183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:25.561280 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.561208 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 17:08:25.561280 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.561258 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:25.562068 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.562000 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 17:08:25.562877 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.562178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.562877 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.562660 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xnsvg\"" Jul 28 17:08:25.562877 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.562661 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fcnc9\"" Jul 28 17:08:25.564227 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.563723 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 17:08:25.564227 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.563878 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:25.564227 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.564005 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:25.564227 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.564126 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 17:08:25.564574 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.564559 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 17:08:25.564896 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.564880 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 17:08:25.565311 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.565294 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 17:08:25.565628 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.565610 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dq76s\"" Jul 28 17:08:25.565771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.565751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.566761 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.566418 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.567857 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.567835 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.568883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/207e36ae-5860-4aa8-841b-a3a8ce36da7d-host\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.568924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-system-cni-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.568950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-cni-multus\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.568976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-systemd-units\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.568999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-run-netns\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9nvb\" (UniqueName: \"kubernetes.io/projected/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-kube-api-access-x9nvb\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-k8s-cni-cncf-io\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-cni-bin\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-kubelet\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-slash\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.569239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-ovn\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.569885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-cni-netd\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.569885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.569885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-env-overrides\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.569885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-socket-dir-parent\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-daemon-config\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-etc-kubernetes\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.569885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtv2\" (UniqueName: \"kubernetes.io/projected/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-kube-api-access-mhtv2\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-tmp-dir\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569955 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zbt9\" (UniqueName: \"kubernetes.io/projected/207e36ae-5860-4aa8-841b-a3a8ce36da7d-kube-api-access-2zbt9\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.569979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-cni-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-os-release\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-hosts-file\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-systemd\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-etc-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-cni-bin\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovnkube-config\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-cnibin\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-hostroot\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570259 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-kubelet\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-node-log\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovn-node-metrics-cert\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-kubelet-config\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570398 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-dbus\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-cni-binary-copy\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svgq7\" (UniqueName: \"kubernetes.io/projected/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-kube-api-access-svgq7\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-log-socket\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/207e36ae-5860-4aa8-841b-a3a8ce36da7d-serviceca\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-netns\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-conf-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570609 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-var-lib-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovnkube-script-lib\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-multus-certs\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.570792 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.570669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.571948 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.571733 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 17:08:25.576834 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.576809 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 17:08:25.577246 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.577225 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 17:08:25.577361 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.577279 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 17:08:25.577488 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.577451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 17:08:25.577663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.577643 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wkj9g\"" Jul 28 17:08:25.577754 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.577737 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-87jcl\"" Jul 28 17:08:25.578452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.578434 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wnq5m\"" Jul 28 17:08:25.578541 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.578456 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 17:08:25.578759 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.578741 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.595376 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.595424 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.595815 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.595935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4zv65\"" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.596021 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.596431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.596529 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 17:08:25.597137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.596887 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 17:08:25.597653 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.597158 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-dfqt4\"" Jul 28 17:08:25.598727 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.598698 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 17:08:25.614958 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.614929 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2vtnw" Jul 28 17:08:25.626232 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.626201 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2vtnw" Jul 28 17:08:25.660363 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.660321 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 17:08:25.671341 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-cni-binary-copy\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-var-lib-kubelet\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-netns\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88902a6f-3b8e-42cc-98d7-806d3958516b-host-slash\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-tuned\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671443 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/207e36ae-5860-4aa8-841b-a3a8ce36da7d-host\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671485 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-kubernetes\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-systemd\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.671526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671524 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72ec7acd-2da5-4722-8e3a-a17180e80677-tmp\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctj8\" (UniqueName: \"kubernetes.io/projected/72ec7acd-2da5-4722-8e3a-a17180e80677-kube-api-access-qctj8\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a727c9ba-bd47-42de-b962-dfb4560caf3c-konnectivity-ca\") pod \"konnectivity-agent-b585j\" (UID: \"a727c9ba-bd47-42de-b962-dfb4560caf3c\") " pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-system-cni-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-cni-multus\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-slash\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-ovn\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-cni-netd\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rx4v\" (UniqueName: \"kubernetes.io/projected/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-kube-api-access-8rx4v\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-socket-dir-parent\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtv2\" (UniqueName: \"kubernetes.io/projected/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-kube-api-access-mhtv2\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zbt9\" (UniqueName: \"kubernetes.io/projected/207e36ae-5860-4aa8-841b-a3a8ce36da7d-kube-api-access-2zbt9\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-modprobe-d\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.671947 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-lib-modules\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-socket-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.671992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-cni-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-systemd\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-cni-bin\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovnkube-config\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672162 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88902a6f-3b8e-42cc-98d7-806d3958516b-iptables-alerter-script\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a727c9ba-bd47-42de-b962-dfb4560caf3c-agent-certs\") pod \"konnectivity-agent-b585j\" (UID: \"a727c9ba-bd47-42de-b962-dfb4560caf3c\") " pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-etc-selinux\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-cnibin\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-node-log\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-kubelet-config\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svgq7\" (UniqueName: \"kubernetes.io/projected/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-kube-api-access-svgq7\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-log-socket\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.672805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/207e36ae-5860-4aa8-841b-a3a8ce36da7d-serviceca\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cnibin\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-os-release\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6txv\" (UniqueName: \"kubernetes.io/projected/9317bc9f-8391-418f-b1dc-4f21e533a8da-kube-api-access-l6txv\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-cni-binary-copy\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-conf-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-var-lib-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-conf-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-system-cni-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-cni-multus\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovnkube-script-lib\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-cni-dir\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysconfig\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysctl-d\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.672988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-registration-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-multus-certs\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.673548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-host\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-sys-fs\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-systemd-units\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-run-netns\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9nvb\" (UniqueName: \"kubernetes.io/projected/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-kube-api-access-x9nvb\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7ph\" (UniqueName: \"kubernetes.io/projected/fa1f8243-02fc-4d70-b1e6-d14221273d8a-kube-api-access-st7ph\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tz6x\" (UniqueName: \"kubernetes.io/projected/88902a6f-3b8e-42cc-98d7-806d3958516b-kube-api-access-6tz6x\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-k8s-cni-cncf-io\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673332 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-cni-bin\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-kubelet\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-env-overrides\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysctl-conf\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-daemon-config\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-etc-kubernetes\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-tmp-dir\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.674339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-os-release\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-hosts-file\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-etc-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-dbus\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-run\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-sys\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-hostroot\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-kubelet\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovn-node-metrics-cert\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.673956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-device-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-multus-certs\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674085 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-systemd-units\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674168 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-run-netns\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-env-overrides\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674373 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-var-lib-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-slash\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674447 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-ovn\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674490 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-k8s-cni-cncf-io\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.674512 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-cni-bin\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-var-lib-kubelet\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.674591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret podName:5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:26.174557057 +0000 UTC m=+2.978245642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret") pod "global-pull-secret-syncer-djz6z" (UID: "5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-host-run-netns\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-cni-netd\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-socket-dir-parent\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.674770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/207e36ae-5860-4aa8-841b-a3a8ce36da7d-host\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-multus-daemon-config\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-etc-kubernetes\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-tmp-dir\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-os-release\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675522 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-hosts-file\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-etc-openvswitch\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-run-ovn-kubernetes\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.675965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675667 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-dbus\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675669 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovnkube-script-lib\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-kubelet\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-hostroot\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.675764 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-cni-bin\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovnkube-config\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676243 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676259 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-cnibin\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-kubelet-config\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676608 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-log-socket\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-run-systemd\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676668 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-node-log\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.677412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.676762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/207e36ae-5860-4aa8-841b-a3a8ce36da7d-serviceca\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.681414 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.680605 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-ovn-node-metrics-cert\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.685421 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.685367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" event={"ID":"df396ec83974a4b79253cc8ed0dbbedd","Type":"ContainerStarted","Data":"82dc6ea9f1dac21c5dfeff7e15b03d692cc90342aa9630eb215b742a45b67b37"} Jul 28 17:08:25.686760 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.686728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" event={"ID":"4fcf926d021c29c5f806efa917b417f7","Type":"ContainerStarted","Data":"c58f5273c3f61cf637e6ed02bd6259d4c2573f04d6cc9ef5cff4bfd43f786868"} Jul 28 17:08:25.691636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.691606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zbt9\" (UniqueName: \"kubernetes.io/projected/207e36ae-5860-4aa8-841b-a3a8ce36da7d-kube-api-access-2zbt9\") pod \"node-ca-6clsf\" (UID: \"207e36ae-5860-4aa8-841b-a3a8ce36da7d\") " pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.693624 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.693567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9nvb\" (UniqueName: \"kubernetes.io/projected/827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f-kube-api-access-x9nvb\") pod \"ovnkube-node-lbjhs\" (UID: \"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f\") " pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.694871 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.694846 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svgq7\" (UniqueName: \"kubernetes.io/projected/a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49-kube-api-access-svgq7\") pod \"node-resolver-v9vdk\" (UID: \"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49\") " pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.695299 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.695279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtv2\" (UniqueName: \"kubernetes.io/projected/207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e-kube-api-access-mhtv2\") pod \"multus-9ntls\" (UID: \"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e\") " pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.775271 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysctl-conf\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775271 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775316 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-run\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-sys\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-device-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-var-lib-kubelet\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88902a6f-3b8e-42cc-98d7-806d3958516b-host-slash\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-tuned\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-kubernetes\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-systemd\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72ec7acd-2da5-4722-8e3a-a17180e80677-tmp\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctj8\" (UniqueName: \"kubernetes.io/projected/72ec7acd-2da5-4722-8e3a-a17180e80677-kube-api-access-qctj8\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.775612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a727c9ba-bd47-42de-b962-dfb4560caf3c-konnectivity-ca\") pod \"konnectivity-agent-b585j\" (UID: \"a727c9ba-bd47-42de-b962-dfb4560caf3c\") " pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rx4v\" (UniqueName: \"kubernetes.io/projected/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-kube-api-access-8rx4v\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-modprobe-d\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-lib-modules\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-socket-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88902a6f-3b8e-42cc-98d7-806d3958516b-iptables-alerter-script\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a727c9ba-bd47-42de-b962-dfb4560caf3c-agent-certs\") pod \"konnectivity-agent-b585j\" (UID: \"a727c9ba-bd47-42de-b962-dfb4560caf3c\") " pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-etc-selinux\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cnibin\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-os-release\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.775998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6txv\" (UniqueName: \"kubernetes.io/projected/9317bc9f-8391-418f-b1dc-4f21e533a8da-kube-api-access-l6txv\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.776070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.776841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysconfig\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.776841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776179 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-sys\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.776841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776262 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88902a6f-3b8e-42cc-98d7-806d3958516b-host-slash\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.776841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776289 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-lib-modules\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.776841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-device-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.776841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776370 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-var-lib-kubelet\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.776841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776712 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.777190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.776953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/88902a6f-3b8e-42cc-98d7-806d3958516b-iptables-alerter-script\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.777190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-modprobe-d\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.777190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-socket-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.777190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-kubernetes\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.777372 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.777372 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysctl-conf\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.777474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysctl-d\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.777474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-system-cni-dir\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.777474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-registration-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.777474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-etc-selinux\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777475 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-os-release\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-host\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-sys-fs\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-host\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cnibin\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7ph\" (UniqueName: \"kubernetes.io/projected/fa1f8243-02fc-4d70-b1e6-d14221273d8a-kube-api-access-st7ph\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tz6x\" (UniqueName: \"kubernetes.io/projected/88902a6f-3b8e-42cc-98d7-806d3958516b-kube-api-access-6tz6x\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.777663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-registration-dir\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.778103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777736 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysctl-d\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.778103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777936 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-sysconfig\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.778103 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.777981 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-systemd\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.778305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.778134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a727c9ba-bd47-42de-b962-dfb4560caf3c-konnectivity-ca\") pod \"konnectivity-agent-b585j\" (UID: \"a727c9ba-bd47-42de-b962-dfb4560caf3c\") " pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.778305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.778201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9317bc9f-8391-418f-b1dc-4f21e533a8da-sys-fs\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.778305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.778260 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72ec7acd-2da5-4722-8e3a-a17180e80677-run\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.778305 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.778262 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:25.778485 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.778331 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs podName:57d60d9c-cb5f-42a6-8f9c-9de2c773f26b nodeName:}" failed. No retries permitted until 2026-07-28 17:08:26.278309927 +0000 UTC m=+3.081998517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs") pod "network-metrics-daemon-g2b76" (UID: "57d60d9c-cb5f-42a6-8f9c-9de2c773f26b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:25.778485 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.778332 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.778485 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.778428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.778623 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.778579 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa1f8243-02fc-4d70-b1e6-d14221273d8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.780680 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.780655 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72ec7acd-2da5-4722-8e3a-a17180e80677-tmp\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.780808 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.780761 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72ec7acd-2da5-4722-8e3a-a17180e80677-etc-tuned\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.781940 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.781917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a727c9ba-bd47-42de-b962-dfb4560caf3c-agent-certs\") pod \"konnectivity-agent-b585j\" (UID: \"a727c9ba-bd47-42de-b962-dfb4560caf3c\") " pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.793433 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.793396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7ph\" (UniqueName: \"kubernetes.io/projected/fa1f8243-02fc-4d70-b1e6-d14221273d8a-kube-api-access-st7ph\") pod \"multus-additional-cni-plugins-29j4x\" (UID: \"fa1f8243-02fc-4d70-b1e6-d14221273d8a\") " pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.794318 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.794293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rx4v\" (UniqueName: \"kubernetes.io/projected/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-kube-api-access-8rx4v\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:25.794816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.794791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tz6x\" (UniqueName: \"kubernetes.io/projected/88902a6f-3b8e-42cc-98d7-806d3958516b-kube-api-access-6tz6x\") pod \"iptables-alerter-gvh5c\" (UID: \"88902a6f-3b8e-42cc-98d7-806d3958516b\") " pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:25.796826 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.796794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6txv\" (UniqueName: \"kubernetes.io/projected/9317bc9f-8391-418f-b1dc-4f21e533a8da-kube-api-access-l6txv\") pod \"aws-ebs-csi-driver-node-twlv6\" (UID: \"9317bc9f-8391-418f-b1dc-4f21e533a8da\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.805931 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.805893 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:25.805931 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.805928 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:25.806164 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.805943 2579 projected.go:196] Error preparing data for projected volume kube-api-access-2q2zj for pod openshift-network-diagnostics/network-check-target-2mnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:25.806164 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:25.806036 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj podName:7fd4f7b3-d3df-4c99-b183-a249347f2dd8 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:26.306013952 +0000 UTC m=+3.109702588 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2q2zj" (UniqueName: "kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj") pod "network-check-target-2mnrj" (UID: "7fd4f7b3-d3df-4c99-b183-a249347f2dd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:25.809141 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.808851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctj8\" (UniqueName: \"kubernetes.io/projected/72ec7acd-2da5-4722-8e3a-a17180e80677-kube-api-access-qctj8\") pod \"tuned-hgq9x\" (UID: \"72ec7acd-2da5-4722-8e3a-a17180e80677\") " pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.868191 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.868150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v9vdk" Jul 28 17:08:25.876663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.876629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:25.887595 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.887559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6clsf" Jul 28 17:08:25.893133 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.893092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9ntls" Jul 28 17:08:25.907054 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.907015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" Jul 28 17:08:25.916024 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.915990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" Jul 28 17:08:25.924985 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.924940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:25.935269 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.935229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-29j4x" Jul 28 17:08:25.941359 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:25.941328 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gvh5c" Jul 28 17:08:26.180634 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.180538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:26.180792 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.180718 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:26.180792 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.180802 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret podName:5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:27.18077557 +0000 UTC m=+3.984464137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret") pod "global-pull-secret-syncer-djz6z" (UID: "5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:26.281699 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.281659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:26.281876 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.281795 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:26.281876 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.281865 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs podName:57d60d9c-cb5f-42a6-8f9c-9de2c773f26b nodeName:}" failed. No retries permitted until 2026-07-28 17:08:27.281845681 +0000 UTC m=+4.085534251 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs") pod "network-metrics-daemon-g2b76" (UID: "57d60d9c-cb5f-42a6-8f9c-9de2c773f26b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:26.319789 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:26.319759 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda727c9ba_bd47_42de_b962_dfb4560caf3c.slice/crio-1971f34346c85b4d7b8807d9ce765037ca162bc078ec57933493fb412d00c07d WatchSource:0}: Error finding container 1971f34346c85b4d7b8807d9ce765037ca162bc078ec57933493fb412d00c07d: Status 404 returned error can't find the container with id 1971f34346c85b4d7b8807d9ce765037ca162bc078ec57933493fb412d00c07d Jul 28 17:08:26.320766 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:26.320736 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b34bff_2ba4_4bf3_8b96_fb67b5bc3c49.slice/crio-bd14db7be9f59381dd8f3f6f3889dbf1560a3f65071bad4d310e37cd8b606860 WatchSource:0}: Error finding container bd14db7be9f59381dd8f3f6f3889dbf1560a3f65071bad4d310e37cd8b606860: Status 404 returned error can't find the container with id bd14db7be9f59381dd8f3f6f3889dbf1560a3f65071bad4d310e37cd8b606860 Jul 28 17:08:26.321910 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:26.321885 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207197ff_dc5a_43b3_a8f3_e3d5f1ddc70e.slice/crio-ba7181dd4ff33b0e08fcf4d8773ab2a5da9feeda18e429a33c1a632aa5fb3d83 WatchSource:0}: Error finding container ba7181dd4ff33b0e08fcf4d8773ab2a5da9feeda18e429a33c1a632aa5fb3d83: Status 404 returned error can't find the container with id ba7181dd4ff33b0e08fcf4d8773ab2a5da9feeda18e429a33c1a632aa5fb3d83 Jul 28 17:08:26.323274 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:26.323244 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ec7acd_2da5_4722_8e3a_a17180e80677.slice/crio-51dcc7779556fd946b5452551d16123ef5334b3623d8ef6c7fcf8ec33e114418 WatchSource:0}: Error finding container 51dcc7779556fd946b5452551d16123ef5334b3623d8ef6c7fcf8ec33e114418: Status 404 returned error can't find the container with id 51dcc7779556fd946b5452551d16123ef5334b3623d8ef6c7fcf8ec33e114418 Jul 28 17:08:26.382568 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.382531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:26.382734 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.382677 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:26.382734 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.382698 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:26.382734 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.382711 2579 projected.go:196] Error preparing data for projected volume kube-api-access-2q2zj for pod openshift-network-diagnostics/network-check-target-2mnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:26.382873 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.382763 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj podName:7fd4f7b3-d3df-4c99-b183-a249347f2dd8 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:27.382750723 +0000 UTC m=+4.186439291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q2zj" (UniqueName: "kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj") pod "network-check-target-2mnrj" (UID: "7fd4f7b3-d3df-4c99-b183-a249347f2dd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:26.628677 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.628627 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:03:25 +0000 UTC" deadline="2028-01-10 02:24:54.71240632 +0000 UTC" Jul 28 17:08:26.628677 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.628672 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12729h16m28.083737313s" Jul 28 17:08:26.646619 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:26.646590 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827fb8dd_cbad_4a6a_9b1e_8c2a216cee3f.slice/crio-1a7ca8a52a859cb0659ddd649e7f425a92e5fbd45754dbc72e2a0e514c2a2169 WatchSource:0}: Error finding container 1a7ca8a52a859cb0659ddd649e7f425a92e5fbd45754dbc72e2a0e514c2a2169: Status 404 returned error can't find the container with id 1a7ca8a52a859cb0659ddd649e7f425a92e5fbd45754dbc72e2a0e514c2a2169 Jul 28 17:08:26.647908 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:26.647881 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1f8243_02fc_4d70_b1e6_d14221273d8a.slice/crio-74c0ecb5bd86076957a1d4459e9b9d3aa373cc4c9a16b46b23e4476ae9c304a6 WatchSource:0}: Error finding container 74c0ecb5bd86076957a1d4459e9b9d3aa373cc4c9a16b46b23e4476ae9c304a6: Status 404 returned error can't find the container with id 74c0ecb5bd86076957a1d4459e9b9d3aa373cc4c9a16b46b23e4476ae9c304a6 Jul 28 17:08:26.648684 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:26.648590 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88902a6f_3b8e_42cc_98d7_806d3958516b.slice/crio-fea8cb151e22def8f44a4853f6ccb3397cf5a0061fb88e7787565409cb037f49 WatchSource:0}: Error finding container fea8cb151e22def8f44a4853f6ccb3397cf5a0061fb88e7787565409cb037f49: Status 404 returned error can't find the container with id fea8cb151e22def8f44a4853f6ccb3397cf5a0061fb88e7787565409cb037f49 Jul 28 17:08:26.679098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.679069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:26.679255 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:26.679223 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:26.695325 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.695216 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerStarted","Data":"74c0ecb5bd86076957a1d4459e9b9d3aa373cc4c9a16b46b23e4476ae9c304a6"} Jul 28 17:08:26.697032 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.696998 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v9vdk" event={"ID":"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49","Type":"ContainerStarted","Data":"bd14db7be9f59381dd8f3f6f3889dbf1560a3f65071bad4d310e37cd8b606860"} Jul 28 17:08:26.698247 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.698217 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" event={"ID":"9317bc9f-8391-418f-b1dc-4f21e533a8da","Type":"ContainerStarted","Data":"27db65a2c43b357559dd965de9826d4aadbdeb391af4a9503fe154f4d7547749"} Jul 28 17:08:26.700977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.700946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6clsf" event={"ID":"207e36ae-5860-4aa8-841b-a3a8ce36da7d","Type":"ContainerStarted","Data":"77b69d1fc9a797ec92d71c36858b155c880eece5d680852729c5d74ffadf58cb"} Jul 28 17:08:26.703364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.703334 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gvh5c" event={"ID":"88902a6f-3b8e-42cc-98d7-806d3958516b","Type":"ContainerStarted","Data":"fea8cb151e22def8f44a4853f6ccb3397cf5a0061fb88e7787565409cb037f49"} Jul 28 17:08:26.705478 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.705444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"1a7ca8a52a859cb0659ddd649e7f425a92e5fbd45754dbc72e2a0e514c2a2169"} Jul 28 17:08:26.706474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.706452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" event={"ID":"72ec7acd-2da5-4722-8e3a-a17180e80677","Type":"ContainerStarted","Data":"51dcc7779556fd946b5452551d16123ef5334b3623d8ef6c7fcf8ec33e114418"} Jul 28 17:08:26.707469 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.707449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9ntls" event={"ID":"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e","Type":"ContainerStarted","Data":"ba7181dd4ff33b0e08fcf4d8773ab2a5da9feeda18e429a33c1a632aa5fb3d83"} Jul 28 17:08:26.708547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:26.708521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b585j" event={"ID":"a727c9ba-bd47-42de-b962-dfb4560caf3c","Type":"ContainerStarted","Data":"1971f34346c85b4d7b8807d9ce765037ca162bc078ec57933493fb412d00c07d"} Jul 28 17:08:27.190317 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.189379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:27.190317 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.189782 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:27.190317 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.189853 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret podName:5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:29.189833564 +0000 UTC m=+5.993522135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret") pod "global-pull-secret-syncer-djz6z" (UID: "5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:27.290532 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.290490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:27.290720 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.290642 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:27.290720 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.290707 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs podName:57d60d9c-cb5f-42a6-8f9c-9de2c773f26b nodeName:}" failed. No retries permitted until 2026-07-28 17:08:29.290687969 +0000 UTC m=+6.094376562 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs") pod "network-metrics-daemon-g2b76" (UID: "57d60d9c-cb5f-42a6-8f9c-9de2c773f26b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:27.391449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.391406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:27.391676 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.391633 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:27.391676 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.391656 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:27.391676 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.391668 2579 projected.go:196] Error preparing data for projected volume kube-api-access-2q2zj for pod openshift-network-diagnostics/network-check-target-2mnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:27.391848 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.391725 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj podName:7fd4f7b3-d3df-4c99-b183-a249347f2dd8 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:29.39170764 +0000 UTC m=+6.195396222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q2zj" (UniqueName: "kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj") pod "network-check-target-2mnrj" (UID: "7fd4f7b3-d3df-4c99-b183-a249347f2dd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:27.629662 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.629598 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:03:25 +0000 UTC" deadline="2028-02-20 21:13:18.652276081 +0000 UTC" Jul 28 17:08:27.629662 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.629660 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13732h4m51.022619978s" Jul 28 17:08:27.680892 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.680857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:27.681079 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.681013 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:27.681468 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.681445 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:27.681590 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:27.681541 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:27.726488 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.725724 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" event={"ID":"df396ec83974a4b79253cc8ed0dbbedd","Type":"ContainerStarted","Data":"4c76a3a4953566aa17a8bbd934c7fc2defcb6fc9fc5bd20225d8d8187ab2d8e5"} Jul 28 17:08:27.747325 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.746479 2579 generic.go:356] "Generic (PLEG): container finished" podID="4fcf926d021c29c5f806efa917b417f7" containerID="1282878c6bf54a7a638882c4116fb662690ca3e54dfdc25649575306016b4bbc" exitCode=0 Jul 28 17:08:27.747325 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.746546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" event={"ID":"4fcf926d021c29c5f806efa917b417f7","Type":"ContainerDied","Data":"1282878c6bf54a7a638882c4116fb662690ca3e54dfdc25649575306016b4bbc"} Jul 28 17:08:27.747325 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:27.747102 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-73.ec2.internal" podStartSLOduration=3.747084702 podStartE2EDuration="3.747084702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:27.746402974 +0000 UTC m=+4.550091566" watchObservedRunningTime="2026-07-28 17:08:27.747084702 +0000 UTC m=+4.550773294" Jul 28 17:08:28.679155 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:28.679119 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:28.679641 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:28.679262 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:28.769653 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:28.769615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" event={"ID":"4fcf926d021c29c5f806efa917b417f7","Type":"ContainerStarted","Data":"c7f877b7fde17ac0a8178c4eb464f2ff68cb384b515834090dbc2cc9659d8ef8"} Jul 28 17:08:29.208904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:29.208287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:29.208904 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.208445 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:29.208904 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.208515 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret podName:5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:33.208495563 +0000 UTC m=+10.012184149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret") pod "global-pull-secret-syncer-djz6z" (UID: "5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:29.309679 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:29.309641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:29.310132 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.309845 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:29.310132 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.309908 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs podName:57d60d9c-cb5f-42a6-8f9c-9de2c773f26b nodeName:}" failed. No retries permitted until 2026-07-28 17:08:33.309890156 +0000 UTC m=+10.113578737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs") pod "network-metrics-daemon-g2b76" (UID: "57d60d9c-cb5f-42a6-8f9c-9de2c773f26b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:29.410716 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:29.410677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:29.410889 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.410867 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:29.410962 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.410897 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:29.410962 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.410914 2579 projected.go:196] Error preparing data for projected volume kube-api-access-2q2zj for pod openshift-network-diagnostics/network-check-target-2mnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:29.411068 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.410979 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj podName:7fd4f7b3-d3df-4c99-b183-a249347f2dd8 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:33.410959082 +0000 UTC m=+10.214647655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q2zj" (UniqueName: "kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj") pod "network-check-target-2mnrj" (UID: "7fd4f7b3-d3df-4c99-b183-a249347f2dd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:29.679797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:29.679291 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:29.679797 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.679444 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:29.679797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:29.679555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:29.679797 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:29.679685 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:30.678837 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:30.678797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:30.679082 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:30.678941 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:31.678965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:31.678838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:31.678965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:31.678838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:31.679507 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:31.678992 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:31.679507 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:31.679065 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:32.679014 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:32.678975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:32.679493 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:32.679134 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:33.246816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:33.246779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:33.247025 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.246933 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:33.247025 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.247009 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret podName:5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:41.246988266 +0000 UTC m=+18.050676849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret") pod "global-pull-secret-syncer-djz6z" (UID: "5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:33.347808 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:33.347727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:33.347996 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.347868 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:33.347996 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.347949 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs podName:57d60d9c-cb5f-42a6-8f9c-9de2c773f26b nodeName:}" failed. No retries permitted until 2026-07-28 17:08:41.347919295 +0000 UTC m=+18.151607885 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs") pod "network-metrics-daemon-g2b76" (UID: "57d60d9c-cb5f-42a6-8f9c-9de2c773f26b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:33.449055 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:33.448426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:33.449055 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.448613 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:33.449055 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.448632 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:33.449055 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.448646 2579 projected.go:196] Error preparing data for projected volume kube-api-access-2q2zj for pod openshift-network-diagnostics/network-check-target-2mnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:33.449055 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.448703 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj podName:7fd4f7b3-d3df-4c99-b183-a249347f2dd8 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:41.448685837 +0000 UTC m=+18.252374421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q2zj" (UniqueName: "kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj") pod "network-check-target-2mnrj" (UID: "7fd4f7b3-d3df-4c99-b183-a249347f2dd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:33.680759 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:33.680088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:33.680759 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.680229 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:33.680759 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:33.680601 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:33.680759 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:33.680707 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:34.678879 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:34.678710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:34.678879 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:34.678861 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:35.679236 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:35.679199 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:35.679761 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:35.679213 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:35.679761 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:35.679351 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:35.679761 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:35.679410 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:36.679008 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:36.678975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:36.679213 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:36.679092 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:37.679167 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:37.679125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:37.679167 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:37.679162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:37.679663 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:37.679290 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:37.679663 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:37.679427 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:38.678508 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:38.678465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:38.678686 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:38.678590 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:39.678585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:39.678550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:39.679079 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:39.678688 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:39.679079 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:39.678754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:39.679079 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:39.678875 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:40.678472 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:40.678440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:40.678662 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:40.678547 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:41.308818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:41.308780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:41.309057 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.308916 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:41.309057 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.308986 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret podName:5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:57.308967249 +0000 UTC m=+34.112655834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret") pod "global-pull-secret-syncer-djz6z" (UID: "5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:41.409928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:41.409888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:41.410133 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.410055 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:41.410198 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.410159 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs podName:57d60d9c-cb5f-42a6-8f9c-9de2c773f26b nodeName:}" failed. No retries permitted until 2026-07-28 17:08:57.410135652 +0000 UTC m=+34.213824234 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs") pod "network-metrics-daemon-g2b76" (UID: "57d60d9c-cb5f-42a6-8f9c-9de2c773f26b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:41.510760 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:41.510715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:41.510963 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.510856 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:41.510963 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.510881 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:41.510963 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.510895 2579 projected.go:196] Error preparing data for projected volume kube-api-access-2q2zj for pod openshift-network-diagnostics/network-check-target-2mnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:41.510963 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.510949 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj podName:7fd4f7b3-d3df-4c99-b183-a249347f2dd8 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:57.510934814 +0000 UTC m=+34.314623386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q2zj" (UniqueName: "kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj") pod "network-check-target-2mnrj" (UID: "7fd4f7b3-d3df-4c99-b183-a249347f2dd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:41.679049 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:41.679012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:41.679514 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.679161 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:41.679514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:41.679219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:41.679514 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:41.679316 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:42.679434 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:42.679392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:42.679854 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:42.679532 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:43.679534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:43.679495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:43.679534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:43.679536 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:43.680052 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:43.679626 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:43.680052 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:43.679706 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:44.679127 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:44.679077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:44.679325 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:44.679230 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:45.679755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.679427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:45.680377 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.679427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:45.680377 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:45.679861 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:45.680377 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:45.679914 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:45.806340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.806212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b585j" event={"ID":"a727c9ba-bd47-42de-b962-dfb4560caf3c","Type":"ContainerStarted","Data":"98f90e28f1cc24273d598fc94a4a3954172f291323acb4a520b980ae0b1bec31"} Jul 28 17:08:45.808766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.808670 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa1f8243-02fc-4d70-b1e6-d14221273d8a" containerID="c1aa6533775a267971f77d7f22343973bb2437d2dff538062dc99278ed618b87" exitCode=0 Jul 28 17:08:45.808766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.808758 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerDied","Data":"c1aa6533775a267971f77d7f22343973bb2437d2dff538062dc99278ed618b87"} Jul 28 17:08:45.811329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.811278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v9vdk" event={"ID":"a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49","Type":"ContainerStarted","Data":"76fe035ec9cf4d5e876199bd8c0bfa28c8485bf8c05805c9d34dac3575c27c32"} Jul 28 17:08:45.813051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.812942 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" event={"ID":"9317bc9f-8391-418f-b1dc-4f21e533a8da","Type":"ContainerStarted","Data":"98f60d4afc2618c156dadda7730912d8d3817feea50c9c64f973ee856cae9351"} Jul 28 17:08:45.814773 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.814747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6clsf" event={"ID":"207e36ae-5860-4aa8-841b-a3a8ce36da7d","Type":"ContainerStarted","Data":"116df581ca994e06bdf42a8b24d8e257cfb153d346b2c000df875f105f7e5904"} Jul 28 17:08:45.819452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.819421 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:08:45.819845 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.819818 2579 generic.go:356] "Generic (PLEG): container finished" podID="827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f" containerID="18c02f1cf8076a4fa4be9df0d3525c3962f001bbb2b1d0b40b23068c49f51209" exitCode=1 Jul 28 17:08:45.819927 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.819859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"8fd81160e167abf7be7b22a2cf67f864d81d2566e5b6293686e07971b9fbda50"} Jul 28 17:08:45.819991 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.819949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"593616ea54426843dbd17bffefe62e0ce7f42188916c3eb961483414f9d7af88"} Jul 28 17:08:45.820050 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.820005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"80709ee49f960e48ee51a34bcbb970d1a46bb98ac846aa310ebd0feaf5115e27"} Jul 28 17:08:45.820097 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.820068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"f089e1f3ba8caad9c3e13fe6509f49b44ad3f14d4f103bd050e1cdb033d34ffa"} Jul 28 17:08:45.820167 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.820136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerDied","Data":"18c02f1cf8076a4fa4be9df0d3525c3962f001bbb2b1d0b40b23068c49f51209"} Jul 28 17:08:45.820218 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.820187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"b570f26dd5502751dac3d679cd5e2163782ad2630bbe9644d636721cc5054837"} Jul 28 17:08:45.821709 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.821660 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" event={"ID":"72ec7acd-2da5-4722-8e3a-a17180e80677","Type":"ContainerStarted","Data":"a53e39bcdad436d64328a5f6b1ebe456a26745380c924be56b9511760fb0502c"} Jul 28 17:08:45.824102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.824073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9ntls" event={"ID":"207197ff-dc5a-43b3-a8f3-e3d5f1ddc70e","Type":"ContainerStarted","Data":"32a6075dceba1aebdc483b7126d45593cd53be34baa6d85b2fc5197b9ff268fb"} Jul 28 17:08:45.829151 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.829073 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-73.ec2.internal" podStartSLOduration=21.829054263 podStartE2EDuration="21.829054263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:28.794072663 +0000 UTC m=+5.597761255" watchObservedRunningTime="2026-07-28 17:08:45.829054263 +0000 UTC m=+22.632742855" Jul 28 17:08:45.830005 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.829967 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b585j" podStartSLOduration=8.23021359 podStartE2EDuration="22.829954226s" totalImagesPullingTime="14.599740636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:45.829277752 +0000 UTC m=+22.632966343" watchObservedRunningTime="2026-07-28 17:08:45.829954226 +0000 UTC m=+22.633642815" Jul 28 17:08:45.870906 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.870840 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v9vdk" podStartSLOduration=4.026963383 podStartE2EDuration="22.870728694s" totalImagesPullingTime="18.843765311s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:45.851552876 +0000 UTC m=+22.655241468" watchObservedRunningTime="2026-07-28 17:08:45.870728694 +0000 UTC m=+22.674417289" Jul 28 17:08:45.871676 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.871637 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6clsf" podStartSLOduration=4.358436713 podStartE2EDuration="22.871622681s" totalImagesPullingTime="18.513185968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:45.870587848 +0000 UTC m=+22.674276439" watchObservedRunningTime="2026-07-28 17:08:45.871622681 +0000 UTC m=+22.675311274" Jul 28 17:08:45.934400 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.934344 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9ntls" podStartSLOduration=3.699338766 podStartE2EDuration="22.934328454s" totalImagesPullingTime="19.234989688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:45.933743653 +0000 UTC m=+22.737432253" watchObservedRunningTime="2026-07-28 17:08:45.934328454 +0000 UTC m=+22.738017043" Jul 28 17:08:45.954832 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:45.954778 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hgq9x" podStartSLOduration=4.112802264 podStartE2EDuration="22.954761636s" totalImagesPullingTime="18.841959372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:45.954160285 +0000 UTC m=+22.757848876" watchObservedRunningTime="2026-07-28 17:08:45.954761636 +0000 UTC m=+22.758450225" Jul 28 17:08:46.669077 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:46.669034 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 17:08:46.678493 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:46.678460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:46.678650 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:46.678585 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:46.829089 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:46.828996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" event={"ID":"9317bc9f-8391-418f-b1dc-4f21e533a8da","Type":"ContainerStarted","Data":"a2e5efe984bd3e80c2aad5843b10f08c6cb1d5f8762a9795a19a1dd0b7216cf1"} Jul 28 17:08:46.830719 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:46.830686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gvh5c" event={"ID":"88902a6f-3b8e-42cc-98d7-806d3958516b","Type":"ContainerStarted","Data":"59c5b6a0732b2ca3d370ca197ba36e65d4e148fbe7f57328000c61d01ee558a3"} Jul 28 17:08:47.520965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.520926 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:47.521616 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.521596 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:47.543944 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.543894 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gvh5c" podStartSLOduration=6.028850278 podStartE2EDuration="24.543873678s" totalImagesPullingTime="18.5150234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:46.854161145 +0000 UTC m=+23.657849734" watchObservedRunningTime="2026-07-28 17:08:47.543873678 +0000 UTC m=+24.347562271" Jul 28 17:08:47.631010 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.630907 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:08:46.669059169Z","UUID":"3553fa89-4237-4f02-98bf-8ea3634b3b0a","Handler":null,"Name":"","Endpoint":""} Jul 28 17:08:47.633486 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.633461 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 17:08:47.633636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.633510 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 17:08:47.678924 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.678891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:47.678924 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.678911 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:47.679202 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:47.679023 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:47.679202 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:47.679187 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:47.832557 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.832527 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:47.832977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:47.832933 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b585j" Jul 28 17:08:48.678796 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:48.678600 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:48.679020 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:48.678898 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:48.837460 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:48.837422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" event={"ID":"9317bc9f-8391-418f-b1dc-4f21e533a8da","Type":"ContainerStarted","Data":"0b169a11207fdd80841482548dc4f84415e66fd37c0e3e8b762a3201a7c5c7cb"} Jul 28 17:08:48.840992 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:48.840963 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:08:48.841464 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:48.841435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"616a11af45e539bbd23f9b629557b1c3ca03508490242be57c8822d90b46e28f"} Jul 28 17:08:48.867993 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:48.867923 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twlv6" podStartSLOduration=4.927134511 podStartE2EDuration="25.867903321s" totalImagesPullingTime="20.94076881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:48.867383404 +0000 UTC m=+25.671071997" watchObservedRunningTime="2026-07-28 17:08:48.867903321 +0000 UTC m=+25.671591913" Jul 28 17:08:49.678787 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:49.678753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:49.679002 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:49.678872 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:49.679002 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:49.678932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:49.679145 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:49.679069 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:50.679263 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.678988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:50.679928 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:50.679279 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:50.848912 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.848877 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:08:50.849313 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.849287 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"f06f98993f0decd7047d821623f3c2b1a8f9e8a2c8b76280cf443de2e6d91dc2"} Jul 28 17:08:50.849553 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.849524 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:50.849654 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.849587 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:50.849743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.849725 2579 scope.go:117] "RemoveContainer" containerID="18c02f1cf8076a4fa4be9df0d3525c3962f001bbb2b1d0b40b23068c49f51209" Jul 28 17:08:50.851234 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.851213 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa1f8243-02fc-4d70-b1e6-d14221273d8a" containerID="88e40946bdb6596b5997e0d0f26476c200fab65be5a205e437e6011c93fea253" exitCode=0 Jul 28 17:08:50.851334 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.851245 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerDied","Data":"88e40946bdb6596b5997e0d0f26476c200fab65be5a205e437e6011c93fea253"} Jul 28 17:08:50.866300 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:50.866273 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:51.679124 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:51.679072 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:51.679428 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:51.679089 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:51.679428 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:51.679239 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:51.679428 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:51.679273 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:51.857778 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:51.857751 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:08:51.858098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:51.858073 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" event={"ID":"827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f","Type":"ContainerStarted","Data":"1e04294d347a1f86d99048bebeaf301662e949e30e52498ea5de15c137c86ed6"} Jul 28 17:08:51.858324 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:51.858311 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:51.873850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:51.873816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:08:51.893362 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:51.893311 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" podStartSLOduration=10.338456603000001 podStartE2EDuration="28.893295249s" totalImagesPullingTime="18.554838646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:08:51.891781143 +0000 UTC m=+28.695469733" watchObservedRunningTime="2026-07-28 17:08:51.893295249 +0000 UTC m=+28.696983838" Jul 28 17:08:52.517669 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.517635 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g2b76"] Jul 28 17:08:52.517837 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.517749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:52.517890 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:52.517851 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:52.520802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.520775 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-djz6z"] Jul 28 17:08:52.520953 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.520910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:52.521043 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:52.521023 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:52.521537 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.521512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2mnrj"] Jul 28 17:08:52.521663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.521642 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:52.521773 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:52.521751 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:52.863221 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.863029 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa1f8243-02fc-4d70-b1e6-d14221273d8a" containerID="6caaf538853d4a36d1302fb2c9b633541d47ed55282c97978a2e8ce5a1310355" exitCode=0 Jul 28 17:08:52.863592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:52.863133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerDied","Data":"6caaf538853d4a36d1302fb2c9b633541d47ed55282c97978a2e8ce5a1310355"} Jul 28 17:08:53.683382 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:53.683350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:53.683580 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:53.683478 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:54.679204 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:54.679175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:54.679861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:54.679174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:54.679861 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:54.679286 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:54.679861 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:54.679352 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:54.870002 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:54.869972 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa1f8243-02fc-4d70-b1e6-d14221273d8a" containerID="59d72938c264422ecdbccd4acfcda284b4c1313cdc88afb5e4fa9f8fb2597e7b" exitCode=0 Jul 28 17:08:54.870002 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:54.870018 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerDied","Data":"59d72938c264422ecdbccd4acfcda284b4c1313cdc88afb5e4fa9f8fb2597e7b"} Jul 28 17:08:55.678958 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:55.678925 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:55.679163 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:55.679050 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-djz6z" podUID="5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42" Jul 28 17:08:56.679117 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:56.679078 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:56.680083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:56.679078 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:56.680083 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:56.679251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g2b76" podUID="57d60d9c-cb5f-42a6-8f9c-9de2c773f26b" Jul 28 17:08:56.680083 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:56.679306 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2mnrj" podUID="7fd4f7b3-d3df-4c99-b183-a249347f2dd8" Jul 28 17:08:57.340774 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.340738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:57.340974 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.340919 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:57.341046 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.341000 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret podName:5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.340976978 +0000 UTC m=+66.144665552 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret") pod "global-pull-secret-syncer-djz6z" (UID: "5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:08:57.441902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.441856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:57.442094 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.442036 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:57.442191 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.442134 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs podName:57d60d9c-cb5f-42a6-8f9c-9de2c773f26b nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.442097008 +0000 UTC m=+66.245785578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs") pod "network-metrics-daemon-g2b76" (UID: "57d60d9c-cb5f-42a6-8f9c-9de2c773f26b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:08:57.522496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.522466 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-73.ec2.internal" event="NodeReady" Jul 28 17:08:57.522670 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.522624 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 17:08:57.543327 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.543278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:57.543509 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.543473 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:08:57.543509 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.543503 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:08:57.543599 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.543519 2579 projected.go:196] Error preparing data for projected volume kube-api-access-2q2zj for pod openshift-network-diagnostics/network-check-target-2mnrj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:57.543599 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.543585 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj podName:7fd4f7b3-d3df-4c99-b183-a249347f2dd8 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.543564546 +0000 UTC m=+66.347253116 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q2zj" (UniqueName: "kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj") pod "network-check-target-2mnrj" (UID: "7fd4f7b3-d3df-4c99-b183-a249347f2dd8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:08:57.594889 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.594793 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79f48cff97-n7hzb"] Jul 28 17:08:57.597577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.597535 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.601992 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.601958 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 17:08:57.602178 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.601959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6dhdb\"" Jul 28 17:08:57.602178 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.601975 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 17:08:57.602293 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.602050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 17:08:57.609247 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.609218 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 17:08:57.612049 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.612021 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vb2ps"] Jul 28 17:08:57.616676 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.616645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:57.618588 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.618561 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lv5jl"] Jul 28 17:08:57.620145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.620094 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4m279\"" Jul 28 17:08:57.620380 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.620163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 17:08:57.620380 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.620094 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 17:08:57.620380 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.620264 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 17:08:57.620529 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.620491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.623478 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.623433 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pz4fk\"" Jul 28 17:08:57.623478 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.623475 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 17:08:57.623687 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.623488 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 17:08:57.627386 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.627330 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f48cff97-n7hzb"] Jul 28 17:08:57.628594 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.628567 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vb2ps"] Jul 28 17:08:57.643401 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.643349 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lv5jl"] Jul 28 17:08:57.678709 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.678671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:08:57.681793 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.681766 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:08:57.745015 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.744974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-installation-pull-secrets\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.745235 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c240724-e4a1-4888-a88c-099f5ecddf3d-config-volume\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.745235 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-bound-sa-token\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.745235 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a19fa65d-1797-4cf1-bfd2-ad05664b46dc-cert\") pod \"ingress-canary-vb2ps\" (UID: \"a19fa65d-1797-4cf1-bfd2-ad05664b46dc\") " pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:57.745235 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-image-registry-private-configuration\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.745413 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fb517aa-9138-4624-88c6-03022bde4791-ca-trust-extracted\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.745413 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kskl\" (UniqueName: \"kubernetes.io/projected/a19fa65d-1797-4cf1-bfd2-ad05664b46dc-kube-api-access-4kskl\") pod \"ingress-canary-vb2ps\" (UID: \"a19fa65d-1797-4cf1-bfd2-ad05664b46dc\") " pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:57.745413 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wskh4\" (UniqueName: \"kubernetes.io/projected/3c240724-e4a1-4888-a88c-099f5ecddf3d-kube-api-access-wskh4\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.745413 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.745601 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c240724-e4a1-4888-a88c-099f5ecddf3d-tmp-dir\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.745601 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.745601 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-registry-certificates\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.745601 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-trusted-ca\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.745601 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.745543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddtd9\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-kube-api-access-ddtd9\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.846375 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fb517aa-9138-4624-88c6-03022bde4791-ca-trust-extracted\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.846375 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kskl\" (UniqueName: \"kubernetes.io/projected/a19fa65d-1797-4cf1-bfd2-ad05664b46dc-kube-api-access-4kskl\") pod \"ingress-canary-vb2ps\" (UID: \"a19fa65d-1797-4cf1-bfd2-ad05664b46dc\") " pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:57.846375 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wskh4\" (UniqueName: \"kubernetes.io/projected/3c240724-e4a1-4888-a88c-099f5ecddf3d-kube-api-access-wskh4\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.846650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.846650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c240724-e4a1-4888-a88c-099f5ecddf3d-tmp-dir\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.846746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.846746 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.846688 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:08:57.846746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-registry-certificates\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.846746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-trusted-ca\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.846763 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls podName:3c240724-e4a1-4888-a88c-099f5ecddf3d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:58.34674342 +0000 UTC m=+35.150431999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls") pod "dns-default-lv5jl" (UID: "3c240724-e4a1-4888-a88c-099f5ecddf3d") : secret "dns-default-metrics-tls" not found Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtd9\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-kube-api-access-ddtd9\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.846821 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.846838 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f48cff97-n7hzb: secret "image-registry-tls" not found Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:57.846897 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls podName:1fb517aa-9138-4624-88c6-03022bde4791 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:58.346878863 +0000 UTC m=+35.150567447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls") pod "image-registry-79f48cff97-n7hzb" (UID: "1fb517aa-9138-4624-88c6-03022bde4791") : secret "image-registry-tls" not found Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-installation-pull-secrets\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846930 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c240724-e4a1-4888-a88c-099f5ecddf3d-config-volume\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.846946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c240724-e4a1-4888-a88c-099f5ecddf3d-tmp-dir\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.847364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846761 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fb517aa-9138-4624-88c6-03022bde4791-ca-trust-extracted\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.847364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.846977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-bound-sa-token\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.847364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.847003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a19fa65d-1797-4cf1-bfd2-ad05664b46dc-cert\") pod \"ingress-canary-vb2ps\" (UID: \"a19fa65d-1797-4cf1-bfd2-ad05664b46dc\") " pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:57.847364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.847042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-image-registry-private-configuration\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.847364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.847326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-registry-certificates\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.847587 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.847568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-trusted-ca\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.847803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.847782 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3c240724-e4a1-4888-a88c-099f5ecddf3d-config-volume\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.851677 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.851649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-installation-pull-secrets\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.851804 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.851645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a19fa65d-1797-4cf1-bfd2-ad05664b46dc-cert\") pod \"ingress-canary-vb2ps\" (UID: \"a19fa65d-1797-4cf1-bfd2-ad05664b46dc\") " pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:57.851804 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.851649 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-image-registry-private-configuration\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.859076 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.859042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wskh4\" (UniqueName: \"kubernetes.io/projected/3c240724-e4a1-4888-a88c-099f5ecddf3d-kube-api-access-wskh4\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:57.859248 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.859145 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kskl\" (UniqueName: \"kubernetes.io/projected/a19fa65d-1797-4cf1-bfd2-ad05664b46dc-kube-api-access-4kskl\") pod \"ingress-canary-vb2ps\" (UID: \"a19fa65d-1797-4cf1-bfd2-ad05664b46dc\") " pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:57.859620 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.859598 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-bound-sa-token\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.860056 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.860035 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtd9\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-kube-api-access-ddtd9\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:57.931340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:57.931300 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vb2ps" Jul 28 17:08:58.351508 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.351465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:58.351678 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.351530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:58.351678 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:58.351624 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:08:58.351678 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:58.351647 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:08:58.351678 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:58.351672 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f48cff97-n7hzb: secret "image-registry-tls" not found Jul 28 17:08:58.351852 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:58.351707 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls podName:3c240724-e4a1-4888-a88c-099f5ecddf3d nodeName:}" failed. No retries permitted until 2026-07-28 17:08:59.351682332 +0000 UTC m=+36.155370903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls") pod "dns-default-lv5jl" (UID: "3c240724-e4a1-4888-a88c-099f5ecddf3d") : secret "dns-default-metrics-tls" not found Jul 28 17:08:58.351852 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:58.351727 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls podName:1fb517aa-9138-4624-88c6-03022bde4791 nodeName:}" failed. No retries permitted until 2026-07-28 17:08:59.35171991 +0000 UTC m=+36.155408481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls") pod "image-registry-79f48cff97-n7hzb" (UID: "1fb517aa-9138-4624-88c6-03022bde4791") : secret "image-registry-tls" not found Jul 28 17:08:58.679170 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.679048 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:08:58.679170 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.679049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:08:58.682743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.682714 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:08:58.683998 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.683973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr5cl\"" Jul 28 17:08:58.684162 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.684077 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hnfw\"" Jul 28 17:08:58.684162 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.684137 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:08:58.684275 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.684221 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:08:58.950222 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:58.950194 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vb2ps"] Jul 28 17:08:58.954994 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:08:58.954965 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19fa65d_1797_4cf1_bfd2_ad05664b46dc.slice/crio-333926481a3ced7c752e5d763e79e5d6d24f4f8796f3d496d2873acf1694ecec WatchSource:0}: Error finding container 333926481a3ced7c752e5d763e79e5d6d24f4f8796f3d496d2873acf1694ecec: Status 404 returned error can't find the container with id 333926481a3ced7c752e5d763e79e5d6d24f4f8796f3d496d2873acf1694ecec Jul 28 17:08:59.358950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:59.358902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:08:59.359161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:59.358965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:08:59.359161 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:59.359083 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:08:59.359301 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:59.359084 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:08:59.359301 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:59.359196 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f48cff97-n7hzb: secret "image-registry-tls" not found Jul 28 17:08:59.359301 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:59.359186 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls podName:3c240724-e4a1-4888-a88c-099f5ecddf3d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:01.359165635 +0000 UTC m=+38.162854220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls") pod "dns-default-lv5jl" (UID: "3c240724-e4a1-4888-a88c-099f5ecddf3d") : secret "dns-default-metrics-tls" not found Jul 28 17:08:59.359301 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:08:59.359243 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls podName:1fb517aa-9138-4624-88c6-03022bde4791 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:01.359232618 +0000 UTC m=+38.162921187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls") pod "image-registry-79f48cff97-n7hzb" (UID: "1fb517aa-9138-4624-88c6-03022bde4791") : secret "image-registry-tls" not found Jul 28 17:08:59.885602 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:59.885558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vb2ps" event={"ID":"a19fa65d-1797-4cf1-bfd2-ad05664b46dc","Type":"ContainerStarted","Data":"333926481a3ced7c752e5d763e79e5d6d24f4f8796f3d496d2873acf1694ecec"} Jul 28 17:08:59.888698 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:59.888660 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa1f8243-02fc-4d70-b1e6-d14221273d8a" containerID="50e43661dafd16da8f2429e06226e6abcb2747b20f7e416342b469f34239fe50" exitCode=0 Jul 28 17:08:59.888698 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:08:59.888706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerDied","Data":"50e43661dafd16da8f2429e06226e6abcb2747b20f7e416342b469f34239fe50"} Jul 28 17:09:00.805342 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:00.805297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:00.895374 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:00.895327 2579 generic.go:356] "Generic (PLEG): container finished" podID="fa1f8243-02fc-4d70-b1e6-d14221273d8a" containerID="a633bada5c2fab78a03b25460c71d9e041e206affc02ba36b62275924f621d38" exitCode=0 Jul 28 17:09:00.895843 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:00.895418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerDied","Data":"a633bada5c2fab78a03b25460c71d9e041e206affc02ba36b62275924f621d38"} Jul 28 17:09:00.952173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:00.952137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:01.375359 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:01.375323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:01.375524 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:01.375428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:01.375524 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:01.375475 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:01.375524 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:01.375495 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f48cff97-n7hzb: secret "image-registry-tls" not found Jul 28 17:09:01.375524 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:01.375523 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:01.375666 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:01.375548 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls podName:1fb517aa-9138-4624-88c6-03022bde4791 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:05.375532229 +0000 UTC m=+42.179220798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls") pod "image-registry-79f48cff97-n7hzb" (UID: "1fb517aa-9138-4624-88c6-03022bde4791") : secret "image-registry-tls" not found Jul 28 17:09:01.375666 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:01.375566 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls podName:3c240724-e4a1-4888-a88c-099f5ecddf3d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:05.375554997 +0000 UTC m=+42.179243589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls") pod "dns-default-lv5jl" (UID: "3c240724-e4a1-4888-a88c-099f5ecddf3d") : secret "dns-default-metrics-tls" not found Jul 28 17:09:01.902639 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:01.902597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-29j4x" event={"ID":"fa1f8243-02fc-4d70-b1e6-d14221273d8a","Type":"ContainerStarted","Data":"d2496d2b7871ac33130f50bb026f2b3dd45084548d837f921e8e1545463e4884"} Jul 28 17:09:01.904359 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:01.904315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vb2ps" event={"ID":"a19fa65d-1797-4cf1-bfd2-ad05664b46dc","Type":"ContainerStarted","Data":"b8b060983124389203d649aa291652fec5d1d5e09a9d65681b2618ad2e641665"} Jul 28 17:09:01.934999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:01.934945 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-29j4x" podStartSLOduration=9.531726709 podStartE2EDuration="38.934926453s" totalImagesPullingTime="29.403199744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:01.933322174 +0000 UTC m=+38.737010787" watchObservedRunningTime="2026-07-28 17:09:01.934926453 +0000 UTC m=+38.738615043" Jul 28 17:09:01.955256 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:01.955203 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vb2ps" podStartSLOduration=2.470903845 podStartE2EDuration="4.955184735s" totalImagesPullingTime="2.48428089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:01.954239169 +0000 UTC m=+38.757927774" watchObservedRunningTime="2026-07-28 17:09:01.955184735 +0000 UTC m=+38.758873324" Jul 28 17:09:02.106797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:02.106705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:02.190276 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:02.190236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:02.288999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:02.288967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:02.357928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:02.357815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:03.714292 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:03.714255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:03.789570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:03.789534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:05.408220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:05.408181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:05.408220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:05.408225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:05.408856 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:05.408370 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:05.408856 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:05.408452 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls podName:3c240724-e4a1-4888-a88c-099f5ecddf3d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:13.408431801 +0000 UTC m=+50.212120387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls") pod "dns-default-lv5jl" (UID: "3c240724-e4a1-4888-a88c-099f5ecddf3d") : secret "dns-default-metrics-tls" not found Jul 28 17:09:05.408856 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:05.408377 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:05.408856 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:05.408471 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f48cff97-n7hzb: secret "image-registry-tls" not found Jul 28 17:09:05.408856 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:05.408522 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls podName:1fb517aa-9138-4624-88c6-03022bde4791 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:13.40850878 +0000 UTC m=+50.212197367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls") pod "image-registry-79f48cff97-n7hzb" (UID: "1fb517aa-9138-4624-88c6-03022bde4791") : secret "image-registry-tls" not found Jul 28 17:09:06.330399 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.330360 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9mhvt"] Jul 28 17:09:06.367299 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.367263 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9mhvt"] Jul 28 17:09:06.367501 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.367410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.371333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.371304 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 17:09:06.372284 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.372256 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k4zqq\"" Jul 28 17:09:06.372455 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.372261 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 17:09:06.372577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.372308 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 17:09:06.372697 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.372350 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 17:09:06.404490 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.404443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:06.514617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.514578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:06.517466 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.517429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-api-access-jpkqn\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.517534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.517480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.517566 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.517538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9846a367-9fc2-4345-8d5c-2fde547331f4-crio-socket\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.517609 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.517577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.517643 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.517612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9846a367-9fc2-4345-8d5c-2fde547331f4-data-volume\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.618754 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.618719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.618905 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.618778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9846a367-9fc2-4345-8d5c-2fde547331f4-data-volume\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.618905 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.618815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-api-access-jpkqn\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.618905 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.618836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.618905 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:06.618886 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:09:06.619148 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:06.618982 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:07.118960043 +0000 UTC m=+43.922648621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : secret "insights-runtime-extractor-tls" not found Jul 28 17:09:06.619148 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:06.618993 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:09:06.619148 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.619045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9846a367-9fc2-4345-8d5c-2fde547331f4-crio-socket\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.619148 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.618903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9846a367-9fc2-4345-8d5c-2fde547331f4-crio-socket\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.619148 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:06.619077 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:07.119057317 +0000 UTC m=+43.922745900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : configmap "kube-rbac-proxy" not found Jul 28 17:09:06.619349 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.619207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9846a367-9fc2-4345-8d5c-2fde547331f4-data-volume\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:06.632425 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:06.632396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkqn\" (UniqueName: \"kubernetes.io/projected/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-api-access-jpkqn\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:07.056480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.056394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:07.098080 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.098047 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4"] Jul 28 17:09:07.109834 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.109810 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" Jul 28 17:09:07.113689 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.113658 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 28 17:09:07.113840 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.113711 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pt46p\"" Jul 28 17:09:07.114994 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.114976 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 28 17:09:07.117868 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.117842 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4"] Jul 28 17:09:07.123100 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.123074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:07.123244 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:07.123195 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:09:07.123244 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.123226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:07.123317 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:07.123298 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:08.123279817 +0000 UTC m=+44.926968384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : configmap "kube-rbac-proxy" not found Jul 28 17:09:07.123359 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:07.123337 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:09:07.123400 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:07.123389 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:08.1233769 +0000 UTC m=+44.927065468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : secret "insights-runtime-extractor-tls" not found Jul 28 17:09:07.204367 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.204334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:07.224253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.224215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjjlr\" (UniqueName: \"kubernetes.io/projected/baa0affd-a44c-4f08-820a-3704d90261f3-kube-api-access-bjjlr\") pod \"migrator-6865c9967-6rwf4\" (UID: \"baa0affd-a44c-4f08-820a-3704d90261f3\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" Jul 28 17:09:07.301226 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.301200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:07.325008 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.324922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjjlr\" (UniqueName: \"kubernetes.io/projected/baa0affd-a44c-4f08-820a-3704d90261f3-kube-api-access-bjjlr\") pod \"migrator-6865c9967-6rwf4\" (UID: \"baa0affd-a44c-4f08-820a-3704d90261f3\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" Jul 28 17:09:07.337472 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.337439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjjlr\" (UniqueName: \"kubernetes.io/projected/baa0affd-a44c-4f08-820a-3704d90261f3-kube-api-access-bjjlr\") pod \"migrator-6865c9967-6rwf4\" (UID: \"baa0affd-a44c-4f08-820a-3704d90261f3\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" Jul 28 17:09:07.419643 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.419599 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" Jul 28 17:09:07.563521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.563484 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4"] Jul 28 17:09:07.566181 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:07.566144 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa0affd_a44c_4f08_820a_3704d90261f3.slice/crio-7c4cef3c0cd761d758d4b2931cbc1d9f6161ba262bf4f1a39630949ed3706c4f WatchSource:0}: Error finding container 7c4cef3c0cd761d758d4b2931cbc1d9f6161ba262bf4f1a39630949ed3706c4f: Status 404 returned error can't find the container with id 7c4cef3c0cd761d758d4b2931cbc1d9f6161ba262bf4f1a39630949ed3706c4f Jul 28 17:09:07.627367 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.627331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:07.790873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.790832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:07.919993 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.919900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" event={"ID":"baa0affd-a44c-4f08-820a-3704d90261f3","Type":"ContainerStarted","Data":"7c4cef3c0cd761d758d4b2931cbc1d9f6161ba262bf4f1a39630949ed3706c4f"} Jul 28 17:09:07.929724 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:07.929700 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v9vdk_a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49/dns-node-resolver/0.log" Jul 28 17:09:08.131188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:08.131149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:08.131366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:08.131232 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:08.131366 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:08.131323 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:09:08.131562 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:08.131407 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:10.13138653 +0000 UTC m=+46.935075099 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : secret "insights-runtime-extractor-tls" not found Jul 28 17:09:08.131562 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:08.131328 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:09:08.131562 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:08.131465 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:10.131448925 +0000 UTC m=+46.935137519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : configmap "kube-rbac-proxy" not found Jul 28 17:09:08.173169 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:08.173060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:08.274943 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:08.274892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:08.731027 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:08.730998 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6clsf_207e36ae-5860-4aa8-841b-a3a8ce36da7d/node-ca/0.log" Jul 28 17:09:08.832124 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:08.832061 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:09.016693 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.016597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:09.729017 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.728979 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vb2ps_a19fa65d-1797-4cf1-bfd2-ad05664b46dc/serve-healthcheck-canary/0.log" Jul 28 17:09:09.748642 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.746662 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-gz2f2"] Jul 28 17:09:09.773086 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.773045 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-gz2f2"] Jul 28 17:09:09.773288 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.773214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:09.776259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.776234 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 28 17:09:09.776399 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.776303 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 28 17:09:09.777504 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.777480 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 28 17:09:09.777616 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.777524 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pb6sf\"" Jul 28 17:09:09.777616 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.777499 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 28 17:09:09.858427 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.858387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:09.925521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.925485 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" event={"ID":"baa0affd-a44c-4f08-820a-3704d90261f3","Type":"ContainerStarted","Data":"7750be51e6078d642287cfd6c502453d3c5f81e87607cacbb11caa937d0ae10d"} Jul 28 17:09:09.925719 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.925542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" event={"ID":"baa0affd-a44c-4f08-820a-3704d90261f3","Type":"ContainerStarted","Data":"93c0a70177ef445f3ef3155ff0929af6b4099fbdf9196ade09311685e86fa57d"} Jul 28 17:09:09.947795 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.947738 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-6rwf4" podStartSLOduration=1.292293434 podStartE2EDuration="2.947720877s" totalImagesPullingTime="1.655427443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:09.947019837 +0000 UTC m=+46.750708427" watchObservedRunningTime="2026-07-28 17:09:09.947720877 +0000 UTC m=+46.751409468" Jul 28 17:09:09.950986 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.950951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prsmr\" (UniqueName: \"kubernetes.io/projected/0f0ec26b-f0cc-443d-9112-4a66907ad100-kube-api-access-prsmr\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:09.951152 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.951016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f0ec26b-f0cc-443d-9112-4a66907ad100-signing-cabundle\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:09.951211 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:09.951163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f0ec26b-f0cc-443d-9112-4a66907ad100-signing-key\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.051986 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.051891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f0ec26b-f0cc-443d-9112-4a66907ad100-signing-cabundle\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.052163 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.052015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f0ec26b-f0cc-443d-9112-4a66907ad100-signing-key\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.052163 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.052043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prsmr\" (UniqueName: \"kubernetes.io/projected/0f0ec26b-f0cc-443d-9112-4a66907ad100-kube-api-access-prsmr\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.052630 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.052601 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0f0ec26b-f0cc-443d-9112-4a66907ad100-signing-cabundle\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.054591 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.054559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:10.054866 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.054842 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0f0ec26b-f0cc-443d-9112-4a66907ad100-signing-key\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.061892 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.061862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsmr\" (UniqueName: \"kubernetes.io/projected/0f0ec26b-f0cc-443d-9112-4a66907ad100-kube-api-access-prsmr\") pod \"service-ca-658d7b7c78-gz2f2\" (UID: \"0f0ec26b-f0cc-443d-9112-4a66907ad100\") " pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.081691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.081649 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" Jul 28 17:09:10.152570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.152512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:10.152750 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.152601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:10.152750 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:10.152652 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:09:10.152750 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:10.152685 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:09:10.152750 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:10.152721 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:14.152702924 +0000 UTC m=+50.956391507 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : secret "insights-runtime-extractor-tls" not found Jul 28 17:09:10.152750 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:10.152743 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:14.152733448 +0000 UTC m=+50.956422016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : configmap "kube-rbac-proxy" not found Jul 28 17:09:10.153120 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.153076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:10.232272 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.232231 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-gz2f2"] Jul 28 17:09:10.234869 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:10.234836 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0ec26b_f0cc_443d_9112_4a66907ad100.slice/crio-5e9f12e1ad36c477f593d17e9d13ec7fa9ee54a81e9c9419298b96e5f5976ac3 WatchSource:0}: Error finding container 5e9f12e1ad36c477f593d17e9d13ec7fa9ee54a81e9c9419298b96e5f5976ac3: Status 404 returned error can't find the container with id 5e9f12e1ad36c477f593d17e9d13ec7fa9ee54a81e9c9419298b96e5f5976ac3 Jul 28 17:09:10.747850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.747815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:10.929769 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:10.929722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" event={"ID":"0f0ec26b-f0cc-443d-9112-4a66907ad100","Type":"ContainerStarted","Data":"5e9f12e1ad36c477f593d17e9d13ec7fa9ee54a81e9c9419298b96e5f5976ac3"} Jul 28 17:09:12.936012 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:12.935974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" event={"ID":"0f0ec26b-f0cc-443d-9112-4a66907ad100","Type":"ContainerStarted","Data":"69af57ecffb710d61aec05930b24bd4878cb7c4571da57c3a1b49c61839ffc64"} Jul 28 17:09:12.965956 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:12.965891 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-gz2f2" podStartSLOduration=1.455935809 podStartE2EDuration="3.965870887s" totalImagesPullingTime="2.509935078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:12.96422997 +0000 UTC m=+49.767918562" watchObservedRunningTime="2026-07-28 17:09:12.965870887 +0000 UTC m=+49.769559509" Jul 28 17:09:12.977202 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:12.977156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:13.482438 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:13.482379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:13.482438 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:13.482430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:13.482707 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:13.482525 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:09:13.482707 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:13.482600 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls podName:3c240724-e4a1-4888-a88c-099f5ecddf3d nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.482585439 +0000 UTC m=+66.286274007 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls") pod "dns-default-lv5jl" (UID: "3c240724-e4a1-4888-a88c-099f5ecddf3d") : secret "dns-default-metrics-tls" not found Jul 28 17:09:13.482707 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:13.482599 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:09:13.482707 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:13.482614 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79f48cff97-n7hzb: secret "image-registry-tls" not found Jul 28 17:09:13.482707 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:13.482655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls podName:1fb517aa-9138-4624-88c6-03022bde4791 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:29.482639583 +0000 UTC m=+66.286328174 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls") pod "image-registry-79f48cff97-n7hzb" (UID: "1fb517aa-9138-4624-88c6-03022bde4791") : secret "image-registry-tls" not found Jul 28 17:09:13.999676 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:13.999633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:14.189029 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:14.188987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:14.189260 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:14.189086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:14.189260 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:14.189255 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:09:14.189368 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:09:14.189321 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls podName:9846a367-9fc2-4345-8d5c-2fde547331f4 nodeName:}" failed. No retries permitted until 2026-07-28 17:09:22.189302162 +0000 UTC m=+58.992990736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mhvt" (UID: "9846a367-9fc2-4345-8d5c-2fde547331f4") : secret "insights-runtime-extractor-tls" not found Jul 28 17:09:14.189657 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:14.189632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9846a367-9fc2-4345-8d5c-2fde547331f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:14.237527 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:14.237488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:14.350832 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:14.350773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:14.440951 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:14.440916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:14.697619 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:14.697515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:15.759122 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:15.759073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:16.007750 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:16.007707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:16.146760 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:16.146721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:16.320996 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:16.320961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:16.505100 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:16.504986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:16.696268 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:16.696232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:17.457962 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:17.457913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:17.697835 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:17.697803 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:17.799827 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:17.799725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:18.094568 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:18.094527 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:18.286475 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:18.286440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:18.366152 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:18.366037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:18.487587 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:18.487545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:18.734098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:18.733994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:19.534093 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:19.534060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:19.654284 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:19.654239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:19.762904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:19.762850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:19.854270 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:19.854225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:20.154451 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:20.154354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:20.334664 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:20.334622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:20.519534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:20.519437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:20.731028 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:20.730994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:20.838704 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:20.838671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:20.950736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:20.950697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:21.084484 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:21.084450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:21.413252 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:21.413216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:21.505891 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:21.505858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.102900 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.102870 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.262352 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.262311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:22.264706 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.264681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9846a367-9fc2-4345-8d5c-2fde547331f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mhvt\" (UID: \"9846a367-9fc2-4345-8d5c-2fde547331f4\") " pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:22.278753 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.278691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9mhvt" Jul 28 17:09:22.312764 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.312729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.416124 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.416081 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9mhvt"] Jul 28 17:09:22.418656 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:22.418617 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9846a367_9fc2_4345_8d5c_2fde547331f4.slice/crio-6a8a72415433f70cd41723ea7b7d9d22eaad74a88ae22966ced9c7f2ef1bb98b WatchSource:0}: Error finding container 6a8a72415433f70cd41723ea7b7d9d22eaad74a88ae22966ced9c7f2ef1bb98b: Status 404 returned error can't find the container with id 6a8a72415433f70cd41723ea7b7d9d22eaad74a88ae22966ced9c7f2ef1bb98b Jul 28 17:09:22.428350 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.428311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.617353 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.617261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.724687 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.724647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.833472 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.833424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.958906 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.958817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:22.963680 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:22.963647 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mhvt" event={"ID":"9846a367-9fc2-4345-8d5c-2fde547331f4","Type":"ContainerStarted","Data":"6a8a72415433f70cd41723ea7b7d9d22eaad74a88ae22966ced9c7f2ef1bb98b"} Jul 28 17:09:23.245500 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:23.245403 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:23.377650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:23.377608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:23.831318 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:23.831209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:23.883576 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:23.883544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lbjhs" Jul 28 17:09:23.957436 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:23.957374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:24.500260 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:24.500223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:24.713823 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:24.713785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:24.898467 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:24.898421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:25.102974 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:25.102934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:28.982173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:28.982088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mhvt" event={"ID":"9846a367-9fc2-4345-8d5c-2fde547331f4","Type":"ContainerStarted","Data":"131895a8df1f17ce6b4389cd1fe12a20faa3cf77767048573f08d1c10a41e2c6"} Jul 28 17:09:29.420323 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.420282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:09:29.424267 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.424228 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:09:29.433227 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.433192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42-original-pull-secret\") pod \"global-pull-secret-syncer-djz6z\" (UID: \"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42\") " pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:09:29.489085 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.489043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-djz6z" Jul 28 17:09:29.521561 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.521517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:29.521739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.521585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:29.521739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.521644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:09:29.524386 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.524355 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3c240724-e4a1-4888-a88c-099f5ecddf3d-metrics-tls\") pod \"dns-default-lv5jl\" (UID: \"3c240724-e4a1-4888-a88c-099f5ecddf3d\") " pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:29.524386 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.524381 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"image-registry-79f48cff97-n7hzb\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:29.524952 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.524933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:09:29.535079 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.535045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57d60d9c-cb5f-42a6-8f9c-9de2c773f26b-metrics-certs\") pod \"network-metrics-daemon-g2b76\" (UID: \"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b\") " pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:09:29.600188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.600152 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hnfw\"" Jul 28 17:09:29.608073 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.608037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g2b76" Jul 28 17:09:29.622656 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.622625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:09:29.648337 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.648295 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-djz6z"] Jul 28 17:09:29.705802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.705764 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:09:29.713901 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.713866 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6dhdb\"" Jul 28 17:09:29.716461 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.716439 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:09:29.719957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.719917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:29.726091 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.726065 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2zj\" (UniqueName: \"kubernetes.io/projected/7fd4f7b3-d3df-4c99-b183-a249347f2dd8-kube-api-access-2q2zj\") pod \"network-check-target-2mnrj\" (UID: \"7fd4f7b3-d3df-4c99-b183-a249347f2dd8\") " pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:09:29.740561 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.740533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pz4fk\"" Jul 28 17:09:29.748573 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.748534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:29.763660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.763630 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g2b76"] Jul 28 17:09:29.893550 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.893515 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hr5cl\"" Jul 28 17:09:29.901405 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:29.901349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:09:30.259192 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.258686 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:30.733446 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.733414 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp"] Jul 28 17:09:30.741130 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.741075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:30.845803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.845762 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp"] Jul 28 17:09:30.845803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.845784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:30.846059 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.845826 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-kh899"] Jul 28 17:09:30.850360 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.850330 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 17:09:30.851456 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.851407 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 28 17:09:30.851456 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.851443 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 28 17:09:30.851776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.851496 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 28 17:09:30.851776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.851494 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 17:09:30.852932 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.852895 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 17:09:30.853655 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.853634 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 28 17:09:30.859268 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.859193 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:30.876786 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.876752 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-kh899"] Jul 28 17:09:30.876961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.876886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-kh899" Jul 28 17:09:30.880889 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.880863 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-t6f94\"" Jul 28 17:09:30.881068 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.880941 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 17:09:30.882049 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.882029 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 17:09:30.933408 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.933368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-hub\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:30.933408 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.933408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp9lp\" (UniqueName: \"kubernetes.io/projected/278364f5-6201-464b-8646-5ee82727089f-kube-api-access-sp9lp\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:30.933766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.933433 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/278364f5-6201-464b-8646-5ee82727089f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:30.933766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.933501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:30.933766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.933612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6nhg\" (UniqueName: \"kubernetes.io/projected/f653f775-cac5-44a4-9d6b-aa2ed0866f34-kube-api-access-w6nhg\") pod \"downloads-7c97dc4555-kh899\" (UID: \"f653f775-cac5-44a4-9d6b-aa2ed0866f34\") " pod="openshift-console/downloads-7c97dc4555-kh899" Jul 28 17:09:30.933766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.933638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-ca\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:30.933766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:30.933666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.035179 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-hub\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.035179 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp9lp\" (UniqueName: \"kubernetes.io/projected/278364f5-6201-464b-8646-5ee82727089f-kube-api-access-sp9lp\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.035179 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/278364f5-6201-464b-8646-5ee82727089f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.035454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.035454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6nhg\" (UniqueName: \"kubernetes.io/projected/f653f775-cac5-44a4-9d6b-aa2ed0866f34-kube-api-access-w6nhg\") pod \"downloads-7c97dc4555-kh899\" (UID: \"f653f775-cac5-44a4-9d6b-aa2ed0866f34\") " pod="openshift-console/downloads-7c97dc4555-kh899" Jul 28 17:09:31.035454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-ca\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.035454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.035944 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.035914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/278364f5-6201-464b-8646-5ee82727089f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.037789 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.037767 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.037870 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.037785 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.037981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.037965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-ca\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.038031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.038013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/278364f5-6201-464b-8646-5ee82727089f-hub\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.046797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.046762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:31.050437 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.050403 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp9lp\" (UniqueName: \"kubernetes.io/projected/278364f5-6201-464b-8646-5ee82727089f-kube-api-access-sp9lp\") pod \"cluster-proxy-proxy-agent-7f6b9ff949-67vfp\" (UID: \"278364f5-6201-464b-8646-5ee82727089f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.050734 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.050718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6nhg\" (UniqueName: \"kubernetes.io/projected/f653f775-cac5-44a4-9d6b-aa2ed0866f34-kube-api-access-w6nhg\") pod \"downloads-7c97dc4555-kh899\" (UID: \"f653f775-cac5-44a4-9d6b-aa2ed0866f34\") " pod="openshift-console/downloads-7c97dc4555-kh899" Jul 28 17:09:31.144724 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:31.144679 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d60d9c_cb5f_42a6_8f9c_9de2c773f26b.slice/crio-fa4a2c80a7aec661270b37446dc2caf38403fba317df6125c56c7cd481a9e6c3 WatchSource:0}: Error finding container fa4a2c80a7aec661270b37446dc2caf38403fba317df6125c56c7cd481a9e6c3: Status 404 returned error can't find the container with id fa4a2c80a7aec661270b37446dc2caf38403fba317df6125c56c7cd481a9e6c3 Jul 28 17:09:31.145315 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:31.145293 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6c9d40_ea89_4929_9a71_eb0ca0a4ff42.slice/crio-35a095f006c39f0108267b80fc60a62b9dc7b3622a0cb485d0ee42db4a3b4acb WatchSource:0}: Error finding container 35a095f006c39f0108267b80fc60a62b9dc7b3622a0cb485d0ee42db4a3b4acb: Status 404 returned error can't find the container with id 35a095f006c39f0108267b80fc60a62b9dc7b3622a0cb485d0ee42db4a3b4acb Jul 28 17:09:31.153603 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.153575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:31.163977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.163950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" Jul 28 17:09:31.186539 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.186513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-kh899" Jul 28 17:09:31.254237 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:31.254199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:32.001936 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:32.001822 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g2b76" event={"ID":"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b","Type":"ContainerStarted","Data":"fa4a2c80a7aec661270b37446dc2caf38403fba317df6125c56c7cd481a9e6c3"} Jul 28 17:09:32.003870 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:32.003759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-djz6z" event={"ID":"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42","Type":"ContainerStarted","Data":"35a095f006c39f0108267b80fc60a62b9dc7b3622a0cb485d0ee42db4a3b4acb"} Jul 28 17:09:32.353874 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:32.353835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2mnrj"] Jul 28 17:09:32.363617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:32.363439 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79f48cff97-n7hzb"] Jul 28 17:09:32.367349 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:32.367267 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb517aa_9138_4624_88c6_03022bde4791.slice/crio-12d8cd2e6b63e78b22266e3e7bee1740f91d5b73298911a8a3d9a248db028910 WatchSource:0}: Error finding container 12d8cd2e6b63e78b22266e3e7bee1740f91d5b73298911a8a3d9a248db028910: Status 404 returned error can't find the container with id 12d8cd2e6b63e78b22266e3e7bee1740f91d5b73298911a8a3d9a248db028910 Jul 28 17:09:32.395914 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:32.395876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-kh899"] Jul 28 17:09:32.400736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:32.400641 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lv5jl"] Jul 28 17:09:32.401732 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:32.401688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp"] Jul 28 17:09:32.404499 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:32.404474 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod278364f5_6201_464b_8646_5ee82727089f.slice/crio-30f862de78806be9ef10a93570b76b304cf820623591805c2e12ffe6fe2ccfc7 WatchSource:0}: Error finding container 30f862de78806be9ef10a93570b76b304cf820623591805c2e12ffe6fe2ccfc7: Status 404 returned error can't find the container with id 30f862de78806be9ef10a93570b76b304cf820623591805c2e12ffe6fe2ccfc7 Jul 28 17:09:32.405836 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:32.405808 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c240724_e4a1_4888_a88c_099f5ecddf3d.slice/crio-ee37076ba955dc247d576ffbe94e5d6d01766386d37b83a28ad80116cd83db74 WatchSource:0}: Error finding container ee37076ba955dc247d576ffbe94e5d6d01766386d37b83a28ad80116cd83db74: Status 404 returned error can't find the container with id ee37076ba955dc247d576ffbe94e5d6d01766386d37b83a28ad80116cd83db74 Jul 28 17:09:33.010090 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.010009 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-kh899" event={"ID":"f653f775-cac5-44a4-9d6b-aa2ed0866f34","Type":"ContainerStarted","Data":"8383ac1c81be4c5193841aeaeccb594b319e2cc5d0a9e072745259337b119edb"} Jul 28 17:09:33.015378 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.015340 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2mnrj" event={"ID":"7fd4f7b3-d3df-4c99-b183-a249347f2dd8","Type":"ContainerStarted","Data":"dcc041dba00efa0a4ba5ee4e1fc0782b8d35ae359c934fe3dca0a31789889753"} Jul 28 17:09:33.017582 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.017543 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mhvt" event={"ID":"9846a367-9fc2-4345-8d5c-2fde547331f4","Type":"ContainerStarted","Data":"ecd6c51ed44a1a52be9d2e092a674d0bc62c0333abab0d56a9a0ebf42c829303"} Jul 28 17:09:33.018951 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.018915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" event={"ID":"278364f5-6201-464b-8646-5ee82727089f","Type":"ContainerStarted","Data":"30f862de78806be9ef10a93570b76b304cf820623591805c2e12ffe6fe2ccfc7"} Jul 28 17:09:33.020636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.020609 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" event={"ID":"1fb517aa-9138-4624-88c6-03022bde4791","Type":"ContainerStarted","Data":"814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7"} Jul 28 17:09:33.020751 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.020655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" event={"ID":"1fb517aa-9138-4624-88c6-03022bde4791","Type":"ContainerStarted","Data":"12d8cd2e6b63e78b22266e3e7bee1740f91d5b73298911a8a3d9a248db028910"} Jul 28 17:09:33.020751 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.020739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:33.022002 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.021971 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lv5jl" event={"ID":"3c240724-e4a1-4888-a88c-099f5ecddf3d","Type":"ContainerStarted","Data":"ee37076ba955dc247d576ffbe94e5d6d01766386d37b83a28ad80116cd83db74"} Jul 28 17:09:33.054234 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.054185 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d9bfb9466-mffcz"] Jul 28 17:09:33.061865 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.061814 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" podStartSLOduration=61.061796877 podStartE2EDuration="1m1.061796877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:33.059844044 +0000 UTC m=+69.863532634" watchObservedRunningTime="2026-07-28 17:09:33.061796877 +0000 UTC m=+69.865485469" Jul 28 17:09:33.083459 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.083422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9bfb9466-mffcz"] Jul 28 17:09:33.083636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.083550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.086872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.086718 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 17:09:33.086872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.086718 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 17:09:33.086872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.086726 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5jwjz\"" Jul 28 17:09:33.086872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.086853 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 17:09:33.087260 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.087242 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 17:09:33.087421 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.087397 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 17:09:33.154068 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.153932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brn98\" (UniqueName: \"kubernetes.io/projected/b8d41b58-fb11-4393-b435-b224a15a1553-kube-api-access-brn98\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.154286 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.154077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-service-ca\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.154286 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.154178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-serving-cert\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.154286 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.154221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-oauth-config\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.154286 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.154237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-console-config\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.154534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.154286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-oauth-serving-cert\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.255488 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.255453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brn98\" (UniqueName: \"kubernetes.io/projected/b8d41b58-fb11-4393-b435-b224a15a1553-kube-api-access-brn98\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.255691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.255503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-service-ca\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.255874 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.255849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-serving-cert\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.255977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.255897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-oauth-config\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.255977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.255923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-console-config\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.255977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.255949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-oauth-serving-cert\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.257184 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.256339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-service-ca\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.257184 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.256655 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-oauth-serving-cert\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.257389 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.257300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-console-config\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.290262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.288488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-serving-cert\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.293395 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.293290 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-oauth-config\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.298344 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.298286 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brn98\" (UniqueName: \"kubernetes.io/projected/b8d41b58-fb11-4393-b435-b224a15a1553-kube-api-access-brn98\") pod \"console-d9bfb9466-mffcz\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.395193 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.395145 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:33.634772 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:33.634699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d9bfb9466-mffcz"] Jul 28 17:09:33.640648 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:33.640469 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d41b58_fb11_4393_b435_b224a15a1553.slice/crio-97452a85d8b3681e9d7ec4f6bef2e85f20c39c5664b92e9cb3de2c3b21768290 WatchSource:0}: Error finding container 97452a85d8b3681e9d7ec4f6bef2e85f20c39c5664b92e9cb3de2c3b21768290: Status 404 returned error can't find the container with id 97452a85d8b3681e9d7ec4f6bef2e85f20c39c5664b92e9cb3de2c3b21768290 Jul 28 17:09:34.031511 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.031472 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bfb9466-mffcz" event={"ID":"b8d41b58-fb11-4393-b435-b224a15a1553","Type":"ContainerStarted","Data":"97452a85d8b3681e9d7ec4f6bef2e85f20c39c5664b92e9cb3de2c3b21768290"} Jul 28 17:09:34.035390 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.035270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g2b76" event={"ID":"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b","Type":"ContainerStarted","Data":"a55009104cfb985adaae38a96bd141b7bc1a3cb4041146d0e45508f4a863e826"} Jul 28 17:09:34.035390 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.035326 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g2b76" event={"ID":"57d60d9c-cb5f-42a6-8f9c-9de2c773f26b","Type":"ContainerStarted","Data":"c71064945f72a103d25835d553a512cce9235425d84c0840fa2c3f75d81dc43e"} Jul 28 17:09:34.060484 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.059861 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g2b76" podStartSLOduration=69.025917346 podStartE2EDuration="1m11.059837397s" totalImagesPullingTime="2.033920051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:34.058684716 +0000 UTC m=+70.862373309" watchObservedRunningTime="2026-07-28 17:09:34.059837397 +0000 UTC m=+70.863525989" Jul 28 17:09:34.246288 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.245316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:34.364660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.364619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:34.446887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.446683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:34.639759 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.638489 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75766b6f89-g2jxs"] Jul 28 17:09:34.641930 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.641235 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.646683 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.646635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:34.663271 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.663239 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75766b6f89-g2jxs"] Jul 28 17:09:34.672609 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.672582 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 17:09:34.733658 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.733524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:34.767925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.767600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-trusted-ca-bundle\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.767925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.767657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-serving-cert\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.767925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.767689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-oauth-serving-cert\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.767925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.767718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-oauth-config\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.767925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.767761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-config\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.767925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.767793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-service-ca\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.767925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.767839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5nph\" (UniqueName: \"kubernetes.io/projected/4412fa73-4804-4a2a-bb27-29bf05dd399a-kube-api-access-h5nph\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.868763 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.868722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-trusted-ca-bundle\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.868961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.868778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-serving-cert\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.868961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.868803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-oauth-serving-cert\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.868961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.868829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-oauth-config\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.868961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.868873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-config\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.868961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.868901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-service-ca\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.868961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.868942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5nph\" (UniqueName: \"kubernetes.io/projected/4412fa73-4804-4a2a-bb27-29bf05dd399a-kube-api-access-h5nph\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.870279 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.870246 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-trusted-ca-bundle\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.877438 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.871272 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-oauth-serving-cert\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.877438 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.871819 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-config\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.877438 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.872349 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-service-ca\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.888481 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.888442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-oauth-config\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.896082 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.896012 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-serving-cert\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.902864 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.902769 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5nph\" (UniqueName: \"kubernetes.io/projected/4412fa73-4804-4a2a-bb27-29bf05dd399a-kube-api-access-h5nph\") pod \"console-75766b6f89-g2jxs\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:34.964555 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:34.964285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:35.178796 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:35.178608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75766b6f89-g2jxs"] Jul 28 17:09:36.949800 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:36.949764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:37.017301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:37.017212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:37.466975 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:37.466919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:37.550260 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:37.550132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:37.883352 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:37.883314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:37.993584 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:37.993520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:38.109798 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:38.109765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:38.211698 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:38.211605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:38.393964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:38.393919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:38.698724 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:38.698677 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4412fa73_4804_4a2a_bb27_29bf05dd399a.slice/crio-6a3b24aac6cc674b97ddc4d4e36a5d8734713a25a8ecf82c571c2fa58a29155f WatchSource:0}: Error finding container 6a3b24aac6cc674b97ddc4d4e36a5d8734713a25a8ecf82c571c2fa58a29155f: Status 404 returned error can't find the container with id 6a3b24aac6cc674b97ddc4d4e36a5d8734713a25a8ecf82c571c2fa58a29155f Jul 28 17:09:38.823558 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:38.823520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:38.932198 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:38.932144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:39.061717 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:39.061633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b6f89-g2jxs" event={"ID":"4412fa73-4804-4a2a-bb27-29bf05dd399a","Type":"ContainerStarted","Data":"6a3b24aac6cc674b97ddc4d4e36a5d8734713a25a8ecf82c571c2fa58a29155f"} Jul 28 17:09:40.246698 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:40.246655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:40.338172 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:40.338083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:44.881546 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:44.881492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:45.004620 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:45.004576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:46.090298 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.090260 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2mnrj" event={"ID":"7fd4f7b3-d3df-4c99-b183-a249347f2dd8","Type":"ContainerStarted","Data":"a66be3b1066a8c4c3f72b1ce0d2cecd26208002ff2088062bbe40f1e47dece26"} Jul 28 17:09:46.091733 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.091709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bfb9466-mffcz" event={"ID":"b8d41b58-fb11-4393-b435-b224a15a1553","Type":"ContainerStarted","Data":"18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a"} Jul 28 17:09:46.093208 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.093186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b6f89-g2jxs" event={"ID":"4412fa73-4804-4a2a-bb27-29bf05dd399a","Type":"ContainerStarted","Data":"9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63"} Jul 28 17:09:46.095292 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.095262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mhvt" event={"ID":"9846a367-9fc2-4345-8d5c-2fde547331f4","Type":"ContainerStarted","Data":"4b8d5068359e39ec8ee4741ea1ab986f6d1afdb5ef366d63866639e697649716"} Jul 28 17:09:46.097159 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.097133 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" event={"ID":"278364f5-6201-464b-8646-5ee82727089f","Type":"ContainerStarted","Data":"aece7fb04bf2661fbba1c2cca95b402dec455fa8b0d2644a91e3af292812cc6d"} Jul 28 17:09:46.097276 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.097172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" event={"ID":"278364f5-6201-464b-8646-5ee82727089f","Type":"ContainerStarted","Data":"6dff121c09cfe7082b27ebd9378d2249429ea640854fd269d10a00309cae329a"} Jul 28 17:09:46.098668 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.098643 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-djz6z" event={"ID":"5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42","Type":"ContainerStarted","Data":"2d09a3fcf5fef21f6a03232a4f7b7ccea6311d5e9d7ed26dfecb1a17aec11745"} Jul 28 17:09:46.100404 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.100384 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lv5jl" event={"ID":"3c240724-e4a1-4888-a88c-099f5ecddf3d","Type":"ContainerStarted","Data":"e576229c9086c09630efbbb4ed66354b2b74a029ba0a4a2b8262e3410bd059bb"} Jul 28 17:09:46.100513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.100419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lv5jl" event={"ID":"3c240724-e4a1-4888-a88c-099f5ecddf3d","Type":"ContainerStarted","Data":"2e5ddac6c5bac76ef8e157e09966eeff00e135223eac84459aac236b0bdc8fef"} Jul 28 17:09:46.100513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.100506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:46.126716 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.126084 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d9bfb9466-mffcz" podStartSLOduration=1.24042989 podStartE2EDuration="13.126060007s" totalImagesPullingTime="11.885630117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:46.124382387 +0000 UTC m=+82.928070976" watchObservedRunningTime="2026-07-28 17:09:46.126060007 +0000 UTC m=+82.929748576" Jul 28 17:09:46.153689 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.153628 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75766b6f89-g2jxs" podStartSLOduration=5.075887901 podStartE2EDuration="12.15360905s" totalImagesPullingTime="7.077721149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:46.152816924 +0000 UTC m=+82.956505515" watchObservedRunningTime="2026-07-28 17:09:46.15360905 +0000 UTC m=+82.957297716" Jul 28 17:09:46.179901 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.179848 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-djz6z" podStartSLOduration=68.805472559 podStartE2EDuration="1m23.179831366s" totalImagesPullingTime="14.374358807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:46.178846241 +0000 UTC m=+82.982534832" watchObservedRunningTime="2026-07-28 17:09:46.179831366 +0000 UTC m=+82.983519957" Jul 28 17:09:46.186982 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.186880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:46.210659 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.210592 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lv5jl" podStartSLOduration=36.103032638 podStartE2EDuration="49.210570916s" totalImagesPullingTime="13.107538278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:46.210142147 +0000 UTC m=+83.013830737" watchObservedRunningTime="2026-07-28 17:09:46.210570916 +0000 UTC m=+83.014259507" Jul 28 17:09:46.262044 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.261978 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9mhvt" podStartSLOduration=17.339012588 podStartE2EDuration="40.261954758s" totalImagesPullingTime="22.92294217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:46.261572052 +0000 UTC m=+83.065260642" watchObservedRunningTime="2026-07-28 17:09:46.261954758 +0000 UTC m=+83.065643351" Jul 28 17:09:46.303840 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.303728 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2mnrj" podStartSLOduration=70.141686045 podStartE2EDuration="1m23.303708671s" totalImagesPullingTime="13.162022626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:46.301266134 +0000 UTC m=+83.104954748" watchObservedRunningTime="2026-07-28 17:09:46.303708671 +0000 UTC m=+83.107397260" Jul 28 17:09:46.306675 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.306633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:46.451328 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:46.451287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:47.107781 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:47.107741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" event={"ID":"278364f5-6201-464b-8646-5ee82727089f","Type":"ContainerStarted","Data":"27fe66d5fcca92cbad5e01c61f794c628d4d440d520f233eaebe542b5e4e5e09"} Jul 28 17:09:47.108733 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:47.108659 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:09:47.185745 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:47.185678 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f6b9ff949-67vfp" podStartSLOduration=3.784630966 podStartE2EDuration="17.18565475s" totalImagesPullingTime="13.401023784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:47.185065897 +0000 UTC m=+83.988754514" watchObservedRunningTime="2026-07-28 17:09:47.18565475 +0000 UTC m=+83.989343344" Jul 28 17:09:47.236002 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:47.235958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:47.372822 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:47.372725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:49.725605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:49.725553 2579 patch_prober.go:28] interesting pod/image-registry-79f48cff97-n7hzb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 28 17:09:49.726097 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:49.725616 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" podUID="1fb517aa-9138-4624-88c6-03022bde4791" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:09:50.339929 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:50.339838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:50.439875 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:50.439827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:50.539369 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:50.539321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:52.348450 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.348410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:52.538879 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.538841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:52.643041 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.642957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:52.662074 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.662033 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k8jst"] Jul 28 17:09:52.736978 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.736915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:52.818887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.818849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.822251 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.822218 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5mp2x\"" Jul 28 17:09:52.822426 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.822262 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 17:09:52.823135 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.822973 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 17:09:52.823135 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.823000 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 17:09:52.823811 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.823739 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 17:09:52.823811 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.823781 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 17:09:52.824228 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.824054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 17:09:52.926938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.926846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-metrics-client-ca\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.926938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.926889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.926938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.926924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-wtmp\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.927254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.926968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-tls\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.927254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.927008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-root\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.927254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.927125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjcgq\" (UniqueName: \"kubernetes.io/projected/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-kube-api-access-qjcgq\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.927254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.927168 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-textfile\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.927254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.927194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-sys\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:52.927254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:52.927230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-tls\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-root\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjcgq\" (UniqueName: \"kubernetes.io/projected/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-kube-api-access-qjcgq\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-root\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-textfile\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-sys\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.028963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-metrics-client-ca\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.029381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.028980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.029381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.029018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-wtmp\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.029381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.029256 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-wtmp\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.029381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.029313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-textfile\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.029381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.029323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-sys\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.029850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.029810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-metrics-client-ca\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.029987 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.029810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.032143 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.032065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:53.032607 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.032581 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-tls\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.032719 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.032677 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.039590 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.039553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjcgq\" (UniqueName: \"kubernetes.io/projected/6b394bff-b7cd-4f8c-a141-b5daefcc13e3-kube-api-access-qjcgq\") pod \"node-exporter-k8jst\" (UID: \"6b394bff-b7cd-4f8c-a141-b5daefcc13e3\") " pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.130525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.130487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:53.131195 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.131165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8jst" Jul 28 17:09:53.231415 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.231328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:53.250843 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:53.250812 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b394bff_b7cd_4f8c_a141_b5daefcc13e3.slice/crio-56de8f669e0a015f9c5d7c5e52ddf269545812e0a08e14280795b03192952665 WatchSource:0}: Error finding container 56de8f669e0a015f9c5d7c5e52ddf269545812e0a08e14280795b03192952665: Status 404 returned error can't find the container with id 56de8f669e0a015f9c5d7c5e52ddf269545812e0a08e14280795b03192952665 Jul 28 17:09:53.396315 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.396276 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:53.396727 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.396339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:53.401523 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.401493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:53.612190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.612155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:53.800769 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:53.800736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:54.040363 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.040261 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:09:54.137318 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.137229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jst" event={"ID":"6b394bff-b7cd-4f8c-a141-b5daefcc13e3","Type":"ContainerStarted","Data":"56de8f669e0a015f9c5d7c5e52ddf269545812e0a08e14280795b03192952665"} Jul 28 17:09:54.141005 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.140964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-kh899" event={"ID":"f653f775-cac5-44a4-9d6b-aa2ed0866f34","Type":"ContainerStarted","Data":"fba91278793480b353e18a20584abcb249584317fd5be7a8d50dd5522fe1422a"} Jul 28 17:09:54.141914 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.141656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-kh899" Jul 28 17:09:54.146764 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.146719 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:09:54.153043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.153016 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-kh899" Jul 28 17:09:54.170259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.170100 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-kh899" podStartSLOduration=2.937122308 podStartE2EDuration="24.170082199s" totalImagesPullingTime="21.232959891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:54.167755961 +0000 UTC m=+90.971444548" watchObservedRunningTime="2026-07-28 17:09:54.170082199 +0000 UTC m=+90.973770789" Jul 28 17:09:54.389009 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.388959 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:54.589624 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.589586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:54.712169 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.712053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:54.965649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.965618 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:54.965780 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.965719 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:54.971956 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:54.971931 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:55.147346 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.147295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jst" event={"ID":"6b394bff-b7cd-4f8c-a141-b5daefcc13e3","Type":"ContainerStarted","Data":"aff481f508ceb8e8a0d953309aaf14d237fbe79e2ace4c831787a2caecdec555"} Jul 28 17:09:55.152691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.152650 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:09:55.250301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.250210 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9bfb9466-mffcz"] Jul 28 17:09:55.449728 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.449683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:55.605552 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.605459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:55.716985 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.716923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:55.816520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.816478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:55.975484 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:55.975386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:56.086544 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:56.086505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:56.111177 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:56.111145 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lv5jl" Jul 28 17:09:56.154494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:56.154455 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b394bff-b7cd-4f8c-a141-b5daefcc13e3" containerID="aff481f508ceb8e8a0d953309aaf14d237fbe79e2ace4c831787a2caecdec555" exitCode=0 Jul 28 17:09:56.154678 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:56.154565 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jst" event={"ID":"6b394bff-b7cd-4f8c-a141-b5daefcc13e3","Type":"ContainerDied","Data":"aff481f508ceb8e8a0d953309aaf14d237fbe79e2ace4c831787a2caecdec555"} Jul 28 17:09:56.687770 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:56.687726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:56.773368 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:56.773332 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:57.161314 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.161272 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jst" event={"ID":"6b394bff-b7cd-4f8c-a141-b5daefcc13e3","Type":"ContainerStarted","Data":"793ce2438df236412f92224b780ccccea0492be71a4031d56bee1423ab9ee444"} Jul 28 17:09:57.161592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.161381 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8jst" event={"ID":"6b394bff-b7cd-4f8c-a141-b5daefcc13e3","Type":"ContainerStarted","Data":"bade58a4e226e53c9592618e4491392ec41e4d96160e0773685667da78ed5356"} Jul 28 17:09:57.189301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.189242 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k8jst" podStartSLOduration=3.41138847 podStartE2EDuration="5.189222012s" totalImagesPullingTime="1.67330959s" totalInitContainerRuntime="104.523952ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:09:57.189091816 +0000 UTC m=+93.992780408" watchObservedRunningTime="2026-07-28 17:09:57.189222012 +0000 UTC m=+93.992910602" Jul 28 17:09:57.413564 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.413469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:57.522884 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.522838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:57.612558 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.612518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:57.778439 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.778345 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:57.882150 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:57.882093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:58.019468 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.019416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:58.124311 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.124277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:58.731276 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.731225 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c76bdc7-kz8kj"] Jul 28 17:09:58.751253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.751216 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.753875 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.753842 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c76bdc7-kz8kj"] Jul 28 17:09:58.783907 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.783870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-oauth-config\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.783907 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.783914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-oauth-serving-cert\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.784452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.783939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-serving-cert\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.784452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.784005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bwdq\" (UniqueName: \"kubernetes.io/projected/5336db60-bfa7-44c0-9319-a6b69ff0dd55-kube-api-access-5bwdq\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.784452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.784097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-trusted-ca-bundle\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.784452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.784180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-config\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.784452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.784229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-service-ca\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885012 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.884976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-config\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-service-ca\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-oauth-config\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-oauth-serving-cert\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-serving-cert\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bwdq\" (UniqueName: \"kubernetes.io/projected/5336db60-bfa7-44c0-9319-a6b69ff0dd55-kube-api-access-5bwdq\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885207 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-trusted-ca-bundle\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885889 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-config\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885995 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-service-ca\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.885995 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.885957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-oauth-serving-cert\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.886206 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.886184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-trusted-ca-bundle\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.888950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.888925 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-oauth-config\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.889092 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.888931 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-serving-cert\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:58.901305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:58.901268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bwdq\" (UniqueName: \"kubernetes.io/projected/5336db60-bfa7-44c0-9319-a6b69ff0dd55-kube-api-access-5bwdq\") pod \"console-c76bdc7-kz8kj\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:59.063663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:59.063564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:09:59.128950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:59.128912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:59.242517 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:59.242474 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:09:59.278151 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:09:59.278097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c76bdc7-kz8kj"] Jul 28 17:09:59.281331 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:09:59.281294 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5336db60_bfa7_44c0_9319_a6b69ff0dd55.slice/crio-bc4b3010a4061fcf32d9020d46874659ce335c7427bda73b34c3eeb2a7e712cd WatchSource:0}: Error finding container bc4b3010a4061fcf32d9020d46874659ce335c7427bda73b34c3eeb2a7e712cd: Status 404 returned error can't find the container with id bc4b3010a4061fcf32d9020d46874659ce335c7427bda73b34c3eeb2a7e712cd Jul 28 17:10:00.177991 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:00.177945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c76bdc7-kz8kj" event={"ID":"5336db60-bfa7-44c0-9319-a6b69ff0dd55","Type":"ContainerStarted","Data":"5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f"} Jul 28 17:10:00.178460 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:00.178015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c76bdc7-kz8kj" event={"ID":"5336db60-bfa7-44c0-9319-a6b69ff0dd55","Type":"ContainerStarted","Data":"bc4b3010a4061fcf32d9020d46874659ce335c7427bda73b34c3eeb2a7e712cd"} Jul 28 17:10:00.211368 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:00.211302 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c76bdc7-kz8kj" podStartSLOduration=2.211282402 podStartE2EDuration="2.211282402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:00.210569846 +0000 UTC m=+97.014258437" watchObservedRunningTime="2026-07-28 17:10:00.211282402 +0000 UTC m=+97.014970996" Jul 28 17:10:00.246013 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:00.245956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:00.345636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:00.345590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:00.840198 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:00.840161 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f48cff97-n7hzb"] Jul 28 17:10:00.961525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:00.961493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:01.061394 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:01.061357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:01.163613 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:01.163518 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:07.023664 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:07.023622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:07.161059 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:07.161024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:08.829115 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:08.829084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:08.953070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:08.953027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:09.064457 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.064416 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:10:09.064663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.064493 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:10:09.069730 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.069701 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:10:09.215992 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.215961 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:10:09.297674 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.297637 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75766b6f89-g2jxs"] Jul 28 17:10:09.449840 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.449798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:09.545324 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.545234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:09.647219 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:09.647178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:10.309676 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:10.309638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:10.426381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:10.426342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:18.115233 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:18.115199 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2mnrj" Jul 28 17:10:19.466627 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:19.466590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:19.603848 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:19.603796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:19.684489 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:19.684455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:21.179685 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.179615 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d9bfb9466-mffcz" podUID="b8d41b58-fb11-4393-b435-b224a15a1553" containerName="console" containerID="cri-o://18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a" gracePeriod=15 Jul 28 17:10:21.418060 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.418030 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:10:21.440861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.440769 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-serving-cert\") pod \"b8d41b58-fb11-4393-b435-b224a15a1553\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " Jul 28 17:10:21.440861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.440822 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-oauth-config\") pod \"b8d41b58-fb11-4393-b435-b224a15a1553\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " Jul 28 17:10:21.440861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.440850 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brn98\" (UniqueName: \"kubernetes.io/projected/b8d41b58-fb11-4393-b435-b224a15a1553-kube-api-access-brn98\") pod \"b8d41b58-fb11-4393-b435-b224a15a1553\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " Jul 28 17:10:21.441077 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.441028 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-service-ca\") pod \"b8d41b58-fb11-4393-b435-b224a15a1553\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " Jul 28 17:10:21.441077 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.441061 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-oauth-serving-cert\") pod \"b8d41b58-fb11-4393-b435-b224a15a1553\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " Jul 28 17:10:21.441184 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.441096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-console-config\") pod \"b8d41b58-fb11-4393-b435-b224a15a1553\" (UID: \"b8d41b58-fb11-4393-b435-b224a15a1553\") " Jul 28 17:10:21.441484 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.441443 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-service-ca" (OuterVolumeSpecName: "service-ca") pod "b8d41b58-fb11-4393-b435-b224a15a1553" (UID: "b8d41b58-fb11-4393-b435-b224a15a1553"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:21.441576 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.441503 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b8d41b58-fb11-4393-b435-b224a15a1553" (UID: "b8d41b58-fb11-4393-b435-b224a15a1553"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:21.441576 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.441550 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-console-config" (OuterVolumeSpecName: "console-config") pod "b8d41b58-fb11-4393-b435-b224a15a1553" (UID: "b8d41b58-fb11-4393-b435-b224a15a1553"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:21.443295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.443267 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b8d41b58-fb11-4393-b435-b224a15a1553" (UID: "b8d41b58-fb11-4393-b435-b224a15a1553"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:21.443388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.443347 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d41b58-fb11-4393-b435-b224a15a1553-kube-api-access-brn98" (OuterVolumeSpecName: "kube-api-access-brn98") pod "b8d41b58-fb11-4393-b435-b224a15a1553" (UID: "b8d41b58-fb11-4393-b435-b224a15a1553"). InnerVolumeSpecName "kube-api-access-brn98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:10:21.443445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.443387 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b8d41b58-fb11-4393-b435-b224a15a1553" (UID: "b8d41b58-fb11-4393-b435-b224a15a1553"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:21.542402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.542344 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:21.542402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.542387 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8d41b58-fb11-4393-b435-b224a15a1553-console-oauth-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:21.542402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.542398 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brn98\" (UniqueName: \"kubernetes.io/projected/b8d41b58-fb11-4393-b435-b224a15a1553-kube-api-access-brn98\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:21.542402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.542409 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:21.542402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.542418 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-oauth-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:21.542739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:21.542428 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8d41b58-fb11-4393-b435-b224a15a1553-console-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:22.259031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.258989 2579 generic.go:356] "Generic (PLEG): container finished" podID="b8d41b58-fb11-4393-b435-b224a15a1553" containerID="18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a" exitCode=0 Jul 28 17:10:22.259031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.259030 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d9bfb9466-mffcz" Jul 28 17:10:22.259533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.259057 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bfb9466-mffcz" event={"ID":"b8d41b58-fb11-4393-b435-b224a15a1553","Type":"ContainerDied","Data":"18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a"} Jul 28 17:10:22.259533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.259128 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d9bfb9466-mffcz" event={"ID":"b8d41b58-fb11-4393-b435-b224a15a1553","Type":"ContainerDied","Data":"97452a85d8b3681e9d7ec4f6bef2e85f20c39c5664b92e9cb3de2c3b21768290"} Jul 28 17:10:22.259533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.259144 2579 scope.go:117] "RemoveContainer" containerID="18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a" Jul 28 17:10:22.267210 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.267183 2579 scope.go:117] "RemoveContainer" containerID="18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a" Jul 28 17:10:22.267504 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:10:22.267485 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a\": container with ID starting with 18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a not found: ID does not exist" containerID="18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a" Jul 28 17:10:22.267598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.267514 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a"} err="failed to get container status \"18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a\": rpc error: code = NotFound desc = could not find container \"18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a\": container with ID starting with 18a81e959ae648d83c19e7ae29a129361d53bb55aae3bc9e4bd8a17246c3824a not found: ID does not exist" Jul 28 17:10:22.291706 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.291674 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d9bfb9466-mffcz"] Jul 28 17:10:22.294609 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.294579 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d9bfb9466-mffcz"] Jul 28 17:10:22.922305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:22.922268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:23.687427 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:23.687381 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8d41b58-fb11-4393-b435-b224a15a1553" path="/var/lib/kubelet/pods/b8d41b58-fb11-4393-b435-b224a15a1553/volumes" Jul 28 17:10:23.806805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:23.806768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:23.894556 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:23.894519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:10:25.865708 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:25.865664 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" podUID="1fb517aa-9138-4624-88c6-03022bde4791" containerName="registry" containerID="cri-o://814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7" gracePeriod=30 Jul 28 17:10:26.145421 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.145332 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:10:26.174278 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fb517aa-9138-4624-88c6-03022bde4791-ca-trust-extracted\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174278 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174282 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-trusted-ca\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174300 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174321 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-registry-certificates\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174345 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-installation-pull-secrets\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174369 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddtd9\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-kube-api-access-ddtd9\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174416 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-bound-sa-token\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174451 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-image-registry-private-configuration\") pod \"1fb517aa-9138-4624-88c6-03022bde4791\" (UID: \"1fb517aa-9138-4624-88c6-03022bde4791\") " Jul 28 17:10:26.174829 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174800 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:26.174829 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.174811 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:26.177246 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.177192 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:26.177389 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.177294 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:10:26.177389 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.177343 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:26.177389 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.177367 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:10:26.177555 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.177428 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-kube-api-access-ddtd9" (OuterVolumeSpecName: "kube-api-access-ddtd9") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "kube-api-access-ddtd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:10:26.183871 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.183832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb517aa-9138-4624-88c6-03022bde4791-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1fb517aa-9138-4624-88c6-03022bde4791" (UID: "1fb517aa-9138-4624-88c6-03022bde4791"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:10:26.274728 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.274689 2579 generic.go:356] "Generic (PLEG): container finished" podID="1fb517aa-9138-4624-88c6-03022bde4791" containerID="814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7" exitCode=0 Jul 28 17:10:26.274728 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.274730 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" Jul 28 17:10:26.274977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.274730 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" event={"ID":"1fb517aa-9138-4624-88c6-03022bde4791","Type":"ContainerDied","Data":"814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7"} Jul 28 17:10:26.274977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.274788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79f48cff97-n7hzb" event={"ID":"1fb517aa-9138-4624-88c6-03022bde4791","Type":"ContainerDied","Data":"12d8cd2e6b63e78b22266e3e7bee1740f91d5b73298911a8a3d9a248db028910"} Jul 28 17:10:26.274977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.274809 2579 scope.go:117] "RemoveContainer" containerID="814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275005 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-bound-sa-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275032 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-image-registry-private-configuration\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275050 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fb517aa-9138-4624-88c6-03022bde4791-ca-trust-extracted\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275064 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-trusted-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275077 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-registry-tls\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275091 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fb517aa-9138-4624-88c6-03022bde4791-registry-certificates\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275131 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fb517aa-9138-4624-88c6-03022bde4791-installation-pull-secrets\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.275161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.275147 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddtd9\" (UniqueName: \"kubernetes.io/projected/1fb517aa-9138-4624-88c6-03022bde4791-kube-api-access-ddtd9\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:26.283342 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.283321 2579 scope.go:117] "RemoveContainer" containerID="814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7" Jul 28 17:10:26.283651 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:10:26.283631 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7\": container with ID starting with 814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7 not found: ID does not exist" containerID="814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7" Jul 28 17:10:26.283694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.283660 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7"} err="failed to get container status \"814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7\": rpc error: code = NotFound desc = could not find container \"814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7\": container with ID starting with 814216a2563bb3be241d1341906f44affefeb78cd033e03ed4c573cb5ecc08d7 not found: ID does not exist" Jul 28 17:10:26.311052 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.311012 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79f48cff97-n7hzb"] Jul 28 17:10:26.314123 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:26.314082 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79f48cff97-n7hzb"] Jul 28 17:10:27.685526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:27.685490 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb517aa-9138-4624-88c6-03022bde4791" path="/var/lib/kubelet/pods/1fb517aa-9138-4624-88c6-03022bde4791/volumes" Jul 28 17:10:34.324798 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.324746 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75766b6f89-g2jxs" podUID="4412fa73-4804-4a2a-bb27-29bf05dd399a" containerName="console" containerID="cri-o://9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63" gracePeriod=15 Jul 28 17:10:34.614900 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.614855 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:10:34.639661 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.639624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-oauth-serving-cert\") pod \"4412fa73-4804-4a2a-bb27-29bf05dd399a\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " Jul 28 17:10:34.639873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.639677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5nph\" (UniqueName: \"kubernetes.io/projected/4412fa73-4804-4a2a-bb27-29bf05dd399a-kube-api-access-h5nph\") pod \"4412fa73-4804-4a2a-bb27-29bf05dd399a\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " Jul 28 17:10:34.639873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.639703 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-oauth-config\") pod \"4412fa73-4804-4a2a-bb27-29bf05dd399a\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " Jul 28 17:10:34.639873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.639724 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-service-ca\") pod \"4412fa73-4804-4a2a-bb27-29bf05dd399a\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " Jul 28 17:10:34.640036 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.639939 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-config\") pod \"4412fa73-4804-4a2a-bb27-29bf05dd399a\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " Jul 28 17:10:34.640036 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.639986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-serving-cert\") pod \"4412fa73-4804-4a2a-bb27-29bf05dd399a\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " Jul 28 17:10:34.640036 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.640027 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-trusted-ca-bundle\") pod \"4412fa73-4804-4a2a-bb27-29bf05dd399a\" (UID: \"4412fa73-4804-4a2a-bb27-29bf05dd399a\") " Jul 28 17:10:34.640358 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.640282 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-service-ca" (OuterVolumeSpecName: "service-ca") pod "4412fa73-4804-4a2a-bb27-29bf05dd399a" (UID: "4412fa73-4804-4a2a-bb27-29bf05dd399a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:34.640358 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.640315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4412fa73-4804-4a2a-bb27-29bf05dd399a" (UID: "4412fa73-4804-4a2a-bb27-29bf05dd399a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:34.640494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.640420 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-config" (OuterVolumeSpecName: "console-config") pod "4412fa73-4804-4a2a-bb27-29bf05dd399a" (UID: "4412fa73-4804-4a2a-bb27-29bf05dd399a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:34.640494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.640478 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4412fa73-4804-4a2a-bb27-29bf05dd399a" (UID: "4412fa73-4804-4a2a-bb27-29bf05dd399a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:10:34.642783 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.642750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4412fa73-4804-4a2a-bb27-29bf05dd399a-kube-api-access-h5nph" (OuterVolumeSpecName: "kube-api-access-h5nph") pod "4412fa73-4804-4a2a-bb27-29bf05dd399a" (UID: "4412fa73-4804-4a2a-bb27-29bf05dd399a"). InnerVolumeSpecName "kube-api-access-h5nph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:10:34.642905 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.642819 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4412fa73-4804-4a2a-bb27-29bf05dd399a" (UID: "4412fa73-4804-4a2a-bb27-29bf05dd399a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:34.643060 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.643034 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4412fa73-4804-4a2a-bb27-29bf05dd399a" (UID: "4412fa73-4804-4a2a-bb27-29bf05dd399a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:10:34.741096 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.741053 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-oauth-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:34.741096 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.741090 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5nph\" (UniqueName: \"kubernetes.io/projected/4412fa73-4804-4a2a-bb27-29bf05dd399a-kube-api-access-h5nph\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:34.741096 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.741101 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-oauth-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:34.741361 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.741138 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:34.741361 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.741151 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:34.741361 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.741162 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4412fa73-4804-4a2a-bb27-29bf05dd399a-console-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:34.741361 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:34.741174 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4412fa73-4804-4a2a-bb27-29bf05dd399a-trusted-ca-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:10:35.308393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.308349 2579 generic.go:356] "Generic (PLEG): container finished" podID="4412fa73-4804-4a2a-bb27-29bf05dd399a" containerID="9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63" exitCode=0 Jul 28 17:10:35.308393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.308397 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75766b6f89-g2jxs" Jul 28 17:10:35.308604 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.308436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b6f89-g2jxs" event={"ID":"4412fa73-4804-4a2a-bb27-29bf05dd399a","Type":"ContainerDied","Data":"9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63"} Jul 28 17:10:35.308604 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.308495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75766b6f89-g2jxs" event={"ID":"4412fa73-4804-4a2a-bb27-29bf05dd399a","Type":"ContainerDied","Data":"6a3b24aac6cc674b97ddc4d4e36a5d8734713a25a8ecf82c571c2fa58a29155f"} Jul 28 17:10:35.308604 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.308518 2579 scope.go:117] "RemoveContainer" containerID="9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63" Jul 28 17:10:35.317289 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.317267 2579 scope.go:117] "RemoveContainer" containerID="9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63" Jul 28 17:10:35.317614 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:10:35.317590 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63\": container with ID starting with 9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63 not found: ID does not exist" containerID="9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63" Jul 28 17:10:35.317686 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.317623 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63"} err="failed to get container status \"9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63\": rpc error: code = NotFound desc = could not find container \"9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63\": container with ID starting with 9acf4ac5d17af9668252e4a61357e1de97a4a9e04390aa7e7066f523dc0d4f63 not found: ID does not exist" Jul 28 17:10:35.332892 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.332857 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75766b6f89-g2jxs"] Jul 28 17:10:35.342424 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.342387 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75766b6f89-g2jxs"] Jul 28 17:10:35.683760 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:10:35.683714 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4412fa73-4804-4a2a-bb27-29bf05dd399a" path="/var/lib/kubelet/pods/4412fa73-4804-4a2a-bb27-29bf05dd399a/volumes" Jul 28 17:11:12.217747 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:12.217701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:12.559565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:12.559468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:17.448002 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.447915 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b7bf6658c-d4rld"] Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448168 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb517aa-9138-4624-88c6-03022bde4791" containerName="registry" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448180 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb517aa-9138-4624-88c6-03022bde4791" containerName="registry" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448189 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4412fa73-4804-4a2a-bb27-29bf05dd399a" containerName="console" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448196 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4412fa73-4804-4a2a-bb27-29bf05dd399a" containerName="console" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448206 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8d41b58-fb11-4393-b435-b224a15a1553" containerName="console" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448212 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8d41b58-fb11-4393-b435-b224a15a1553" containerName="console" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448252 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4412fa73-4804-4a2a-bb27-29bf05dd399a" containerName="console" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448260 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8d41b58-fb11-4393-b435-b224a15a1553" containerName="console" Jul 28 17:11:17.448412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.448268 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb517aa-9138-4624-88c6-03022bde4791" containerName="registry" Jul 28 17:11:17.450156 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.450139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.456938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.456600 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 17:11:17.456938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.456906 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tmxqg\"" Jul 28 17:11:17.457074 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.457004 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 17:11:17.458028 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.458008 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 17:11:17.458176 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.458160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 17:11:17.458252 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.458160 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 17:11:17.470858 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.470827 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 17:11:17.548708 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.548672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.548708 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.548708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgzl\" (UniqueName: \"kubernetes.io/projected/68ffba35-1a43-4953-8bc9-c8a99e766b06-kube-api-access-hdgzl\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.548986 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.548732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-telemeter-client-tls\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.548986 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.548750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-metrics-client-ca\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.548986 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.548890 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-federate-client-tls\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.548986 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.548932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.549173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.549008 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-secret-telemeter-client\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.549173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.549061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-serving-certs-ca-bundle\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.552929 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.552902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b7bf6658c-d4rld"] Jul 28 17:11:17.649833 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.649793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-serving-certs-ca-bundle\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.649833 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.649835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.649853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgzl\" (UniqueName: \"kubernetes.io/projected/68ffba35-1a43-4953-8bc9-c8a99e766b06-kube-api-access-hdgzl\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.649907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-telemeter-client-tls\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.649940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-metrics-client-ca\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.650020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-federate-client-tls\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.650045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.650175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-secret-telemeter-client\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650781 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.650755 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-serving-certs-ca-bundle\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650906 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.650802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-metrics-client-ca\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.650906 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.650819 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68ffba35-1a43-4953-8bc9-c8a99e766b06-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.653231 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.653197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-telemeter-client-tls\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.653336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.653197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.653336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.653309 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-federate-client-tls\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.653417 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.653351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/68ffba35-1a43-4953-8bc9-c8a99e766b06-secret-telemeter-client\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.661206 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.661181 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgzl\" (UniqueName: \"kubernetes.io/projected/68ffba35-1a43-4953-8bc9-c8a99e766b06-kube-api-access-hdgzl\") pod \"telemeter-client-7b7bf6658c-d4rld\" (UID: \"68ffba35-1a43-4953-8bc9-c8a99e766b06\") " pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.759640 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.759526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" Jul 28 17:11:17.914726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:17.914631 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b7bf6658c-d4rld"] Jul 28 17:11:17.917378 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:11:17.917349 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ffba35_1a43_4953_8bc9_c8a99e766b06.slice/crio-e1833338c514a0e1bbb04efdb06c46909b41ecfc4bf708133e12981f9b8a3d82 WatchSource:0}: Error finding container e1833338c514a0e1bbb04efdb06c46909b41ecfc4bf708133e12981f9b8a3d82: Status 404 returned error can't find the container with id e1833338c514a0e1bbb04efdb06c46909b41ecfc4bf708133e12981f9b8a3d82 Jul 28 17:11:18.459029 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:18.458992 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" event={"ID":"68ffba35-1a43-4953-8bc9-c8a99e766b06","Type":"ContainerStarted","Data":"e1833338c514a0e1bbb04efdb06c46909b41ecfc4bf708133e12981f9b8a3d82"} Jul 28 17:11:20.468231 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:20.468194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" event={"ID":"68ffba35-1a43-4953-8bc9-c8a99e766b06","Type":"ContainerStarted","Data":"c3adc7c1fc8b19081d5bbf5a58cf6e96c2d8399cd172adcb42cff169843f681f"} Jul 28 17:11:21.474301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:21.474199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" event={"ID":"68ffba35-1a43-4953-8bc9-c8a99e766b06","Type":"ContainerStarted","Data":"06be7a89fbd0a180f6e150d55d0af5ef5475bab68f0d784294d4cb87b11f7c62"} Jul 28 17:11:21.474301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:21.474265 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" event={"ID":"68ffba35-1a43-4953-8bc9-c8a99e766b06","Type":"ContainerStarted","Data":"370d106bd9a963571a044aaec479d4af82a04b7067904969255a4f97f8367057"} Jul 28 17:11:22.270398 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.270325 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b7bf6658c-d4rld" podStartSLOduration=2.120368277 podStartE2EDuration="5.27029976s" totalImagesPullingTime="3.149931483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:21.510506806 +0000 UTC m=+178.314195398" watchObservedRunningTime="2026-07-28 17:11:22.27029976 +0000 UTC m=+179.073988351" Jul 28 17:11:22.270881 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.270864 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-694dbcfdf-87kmr"] Jul 28 17:11:22.273411 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.273381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.290141 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.290088 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-694dbcfdf-87kmr"] Jul 28 17:11:22.340589 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.340549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:22.386516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.386474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-config\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.386646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.386539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccglz\" (UniqueName: \"kubernetes.io/projected/e09d7394-2bea-4d64-bbe8-9de773d0c982-kube-api-access-ccglz\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.386646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.386569 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-service-ca\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.386646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.386592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-trusted-ca-bundle\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.386646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.386614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-oauth-config\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.386776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.386659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-serving-cert\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.386776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.386685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-oauth-serving-cert\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.486993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-oauth-serving-cert\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-config\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccglz\" (UniqueName: \"kubernetes.io/projected/e09d7394-2bea-4d64-bbe8-9de773d0c982-kube-api-access-ccglz\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-service-ca\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-trusted-ca-bundle\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-oauth-config\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487386 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-serving-cert\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.487937 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-oauth-serving-cert\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.488015 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-config\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.488015 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.487914 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-service-ca\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.488176 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.488158 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-trusted-ca-bundle\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.489706 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.489676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-oauth-config\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.489797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.489715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-serving-cert\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.499833 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.499807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccglz\" (UniqueName: \"kubernetes.io/projected/e09d7394-2bea-4d64-bbe8-9de773d0c982-kube-api-access-ccglz\") pod \"console-694dbcfdf-87kmr\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.582824 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.582722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:22.724710 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:22.724677 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-694dbcfdf-87kmr"] Jul 28 17:11:22.727814 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:11:22.727787 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09d7394_2bea_4d64_bbe8_9de773d0c982.slice/crio-99a7686b7d82c4b375551d2f47e449f5863415281534770e6e1b8529d8a0f73b WatchSource:0}: Error finding container 99a7686b7d82c4b375551d2f47e449f5863415281534770e6e1b8529d8a0f73b: Status 404 returned error can't find the container with id 99a7686b7d82c4b375551d2f47e449f5863415281534770e6e1b8529d8a0f73b Jul 28 17:11:23.483178 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:23.483139 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694dbcfdf-87kmr" event={"ID":"e09d7394-2bea-4d64-bbe8-9de773d0c982","Type":"ContainerStarted","Data":"c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e"} Jul 28 17:11:23.483364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:23.483194 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694dbcfdf-87kmr" event={"ID":"e09d7394-2bea-4d64-bbe8-9de773d0c982","Type":"ContainerStarted","Data":"99a7686b7d82c4b375551d2f47e449f5863415281534770e6e1b8529d8a0f73b"} Jul 28 17:11:23.509490 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:23.509434 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-694dbcfdf-87kmr" podStartSLOduration=1.509417001 podStartE2EDuration="1.509417001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:23.507929708 +0000 UTC m=+180.311618298" watchObservedRunningTime="2026-07-28 17:11:23.509417001 +0000 UTC m=+180.313105651" Jul 28 17:11:27.495950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:27.495914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:27.589009 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:27.588962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:28.169559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.169520 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-694dbcfdf-87kmr"] Jul 28 17:11:28.225516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.225479 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77548c74b7-mk95z"] Jul 28 17:11:28.228310 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.228288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.253430 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.253392 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77548c74b7-mk95z"] Jul 28 17:11:28.334732 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.334693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-serving-cert\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.334911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.334738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-config\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.334911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.334770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-oauth-serving-cert\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.334911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.334816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqs6n\" (UniqueName: \"kubernetes.io/projected/718f5f53-35db-42d2-b445-4eccfd22e9d6-kube-api-access-rqs6n\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.334911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.334869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-trusted-ca-bundle\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.334911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.334903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-oauth-config\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.335077 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.334950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-service-ca\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.436085 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.435973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqs6n\" (UniqueName: \"kubernetes.io/projected/718f5f53-35db-42d2-b445-4eccfd22e9d6-kube-api-access-rqs6n\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.436295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.436136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-trusted-ca-bundle\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.436295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.436169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-oauth-config\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.436295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.436189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-service-ca\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.436295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.436222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-serving-cert\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.436295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.436256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-config\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.436540 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.436311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-oauth-serving-cert\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.437521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.437489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-service-ca\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.442802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.437634 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-config\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.442802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.437660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-oauth-serving-cert\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.442802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.437678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-trusted-ca-bundle\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.442802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.439162 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-oauth-config\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.442802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.439726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-serving-cert\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.447838 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.447805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqs6n\" (UniqueName: \"kubernetes.io/projected/718f5f53-35db-42d2-b445-4eccfd22e9d6-kube-api-access-rqs6n\") pod \"console-77548c74b7-mk95z\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.537575 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.537530 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:28.672162 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:28.671975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77548c74b7-mk95z"] Jul 28 17:11:28.674759 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:11:28.674736 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718f5f53_35db_42d2_b445_4eccfd22e9d6.slice/crio-b4c7f26052c0b0befb328582c87807365db281e89f3bb11361d9ad4d65da8720 WatchSource:0}: Error finding container b4c7f26052c0b0befb328582c87807365db281e89f3bb11361d9ad4d65da8720: Status 404 returned error can't find the container with id b4c7f26052c0b0befb328582c87807365db281e89f3bb11361d9ad4d65da8720 Jul 28 17:11:29.505289 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:29.505251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77548c74b7-mk95z" event={"ID":"718f5f53-35db-42d2-b445-4eccfd22e9d6","Type":"ContainerStarted","Data":"8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd"} Jul 28 17:11:29.505468 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:29.505303 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77548c74b7-mk95z" event={"ID":"718f5f53-35db-42d2-b445-4eccfd22e9d6","Type":"ContainerStarted","Data":"b4c7f26052c0b0befb328582c87807365db281e89f3bb11361d9ad4d65da8720"} Jul 28 17:11:29.528157 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:29.528083 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77548c74b7-mk95z" podStartSLOduration=1.528064918 podStartE2EDuration="1.528064918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:29.527884308 +0000 UTC m=+186.331572913" watchObservedRunningTime="2026-07-28 17:11:29.528064918 +0000 UTC m=+186.331753508" Jul 28 17:11:32.583152 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:32.583087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:38.538336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:38.538296 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:38.538855 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:38.538365 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:38.543153 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:38.543124 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:39.359479 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:39.359432 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:39.437603 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:39.437546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:39.544622 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:39.544594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:11:39.604782 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:39.604750 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c76bdc7-kz8kj"] Jul 28 17:11:39.739915 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:39.739822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:39.866732 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:39.866694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:39.942517 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:39.942478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:53.188181 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.188103 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-694dbcfdf-87kmr" podUID="e09d7394-2bea-4d64-bbe8-9de773d0c982" containerName="console" containerID="cri-o://c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e" gracePeriod=15 Jul 28 17:11:53.436373 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.436346 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:53.536950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.536853 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-oauth-serving-cert\") pod \"e09d7394-2bea-4d64-bbe8-9de773d0c982\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " Jul 28 17:11:53.536950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.536903 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccglz\" (UniqueName: \"kubernetes.io/projected/e09d7394-2bea-4d64-bbe8-9de773d0c982-kube-api-access-ccglz\") pod \"e09d7394-2bea-4d64-bbe8-9de773d0c982\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " Jul 28 17:11:53.536950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.536922 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-oauth-config\") pod \"e09d7394-2bea-4d64-bbe8-9de773d0c982\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " Jul 28 17:11:53.537272 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.536977 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-config\") pod \"e09d7394-2bea-4d64-bbe8-9de773d0c982\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " Jul 28 17:11:53.537272 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.537011 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-trusted-ca-bundle\") pod \"e09d7394-2bea-4d64-bbe8-9de773d0c982\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " Jul 28 17:11:53.537272 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.537028 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-service-ca\") pod \"e09d7394-2bea-4d64-bbe8-9de773d0c982\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " Jul 28 17:11:53.537272 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.537075 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-serving-cert\") pod \"e09d7394-2bea-4d64-bbe8-9de773d0c982\" (UID: \"e09d7394-2bea-4d64-bbe8-9de773d0c982\") " Jul 28 17:11:53.537538 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.537505 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-service-ca" (OuterVolumeSpecName: "service-ca") pod "e09d7394-2bea-4d64-bbe8-9de773d0c982" (UID: "e09d7394-2bea-4d64-bbe8-9de773d0c982"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:53.537605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.537510 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e09d7394-2bea-4d64-bbe8-9de773d0c982" (UID: "e09d7394-2bea-4d64-bbe8-9de773d0c982"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:53.537605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.537523 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-config" (OuterVolumeSpecName: "console-config") pod "e09d7394-2bea-4d64-bbe8-9de773d0c982" (UID: "e09d7394-2bea-4d64-bbe8-9de773d0c982"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:53.537605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.537568 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e09d7394-2bea-4d64-bbe8-9de773d0c982" (UID: "e09d7394-2bea-4d64-bbe8-9de773d0c982"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:11:53.539170 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.539144 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09d7394-2bea-4d64-bbe8-9de773d0c982-kube-api-access-ccglz" (OuterVolumeSpecName: "kube-api-access-ccglz") pod "e09d7394-2bea-4d64-bbe8-9de773d0c982" (UID: "e09d7394-2bea-4d64-bbe8-9de773d0c982"). InnerVolumeSpecName "kube-api-access-ccglz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:11:53.539258 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.539177 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e09d7394-2bea-4d64-bbe8-9de773d0c982" (UID: "e09d7394-2bea-4d64-bbe8-9de773d0c982"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:11:53.539258 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.539196 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e09d7394-2bea-4d64-bbe8-9de773d0c982" (UID: "e09d7394-2bea-4d64-bbe8-9de773d0c982"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:11:53.590638 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.590597 2579 generic.go:356] "Generic (PLEG): container finished" podID="e09d7394-2bea-4d64-bbe8-9de773d0c982" containerID="c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e" exitCode=0 Jul 28 17:11:53.590819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.590653 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-694dbcfdf-87kmr" Jul 28 17:11:53.590819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.590697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694dbcfdf-87kmr" event={"ID":"e09d7394-2bea-4d64-bbe8-9de773d0c982","Type":"ContainerDied","Data":"c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e"} Jul 28 17:11:53.590819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.590781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-694dbcfdf-87kmr" event={"ID":"e09d7394-2bea-4d64-bbe8-9de773d0c982","Type":"ContainerDied","Data":"99a7686b7d82c4b375551d2f47e449f5863415281534770e6e1b8529d8a0f73b"} Jul 28 17:11:53.590819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.590803 2579 scope.go:117] "RemoveContainer" containerID="c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e" Jul 28 17:11:53.599675 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.599656 2579 scope.go:117] "RemoveContainer" containerID="c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e" Jul 28 17:11:53.600010 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:11:53.599987 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e\": container with ID starting with c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e not found: ID does not exist" containerID="c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e" Jul 28 17:11:53.600057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.600021 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e"} err="failed to get container status \"c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e\": rpc error: code = NotFound desc = could not find container \"c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e\": container with ID starting with c4fa1b873621b17806f88498079d7a50ffd80651644202bccd8d849c961d6f1e not found: ID does not exist" Jul 28 17:11:53.618390 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.618354 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-694dbcfdf-87kmr"] Jul 28 17:11:53.620070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.620045 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-694dbcfdf-87kmr"] Jul 28 17:11:53.637529 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.637493 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-oauth-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:11:53.637529 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.637525 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccglz\" (UniqueName: \"kubernetes.io/projected/e09d7394-2bea-4d64-bbe8-9de773d0c982-kube-api-access-ccglz\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:11:53.637529 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.637536 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-oauth-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:11:53.637771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.637545 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:11:53.637771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.637554 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-trusted-ca-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:11:53.637771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.637562 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e09d7394-2bea-4d64-bbe8-9de773d0c982-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:11:53.637771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.637573 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e09d7394-2bea-4d64-bbe8-9de773d0c982-console-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:11:53.683852 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.683820 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e09d7394-2bea-4d64-bbe8-9de773d0c982" path="/var/lib/kubelet/pods/e09d7394-2bea-4d64-bbe8-9de773d0c982/volumes" Jul 28 17:11:53.874387 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.874354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:53.975130 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:53.975080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:11:54.063775 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:11:54.063737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:04.623934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.623859 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-c76bdc7-kz8kj" podUID="5336db60-bfa7-44c0-9319-a6b69ff0dd55" containerName="console" containerID="cri-o://5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f" gracePeriod=15 Jul 28 17:12:04.871023 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.870996 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:12:04.913014 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.912932 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-oauth-serving-cert\") pod \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " Jul 28 17:12:04.913014 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.912987 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bwdq\" (UniqueName: \"kubernetes.io/projected/5336db60-bfa7-44c0-9319-a6b69ff0dd55-kube-api-access-5bwdq\") pod \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " Jul 28 17:12:04.913262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913022 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-oauth-config\") pod \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " Jul 28 17:12:04.913262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913047 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-config\") pod \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " Jul 28 17:12:04.913262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913074 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-service-ca\") pod \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " Jul 28 17:12:04.913262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913174 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-trusted-ca-bundle\") pod \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " Jul 28 17:12:04.913507 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913389 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5336db60-bfa7-44c0-9319-a6b69ff0dd55" (UID: "5336db60-bfa7-44c0-9319-a6b69ff0dd55"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:04.913618 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913537 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-config" (OuterVolumeSpecName: "console-config") pod "5336db60-bfa7-44c0-9319-a6b69ff0dd55" (UID: "5336db60-bfa7-44c0-9319-a6b69ff0dd55"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:04.913618 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913566 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-service-ca" (OuterVolumeSpecName: "service-ca") pod "5336db60-bfa7-44c0-9319-a6b69ff0dd55" (UID: "5336db60-bfa7-44c0-9319-a6b69ff0dd55"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:04.913618 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913592 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-serving-cert\") pod \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\" (UID: \"5336db60-bfa7-44c0-9319-a6b69ff0dd55\") " Jul 28 17:12:04.913774 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913619 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5336db60-bfa7-44c0-9319-a6b69ff0dd55" (UID: "5336db60-bfa7-44c0-9319-a6b69ff0dd55"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:12:04.913827 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913786 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-oauth-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:04.913827 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913799 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:04.913827 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913808 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:04.913827 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.913816 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5336db60-bfa7-44c0-9319-a6b69ff0dd55-trusted-ca-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:04.915701 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.915669 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5336db60-bfa7-44c0-9319-a6b69ff0dd55" (UID: "5336db60-bfa7-44c0-9319-a6b69ff0dd55"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:12:04.915818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.915798 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5336db60-bfa7-44c0-9319-a6b69ff0dd55" (UID: "5336db60-bfa7-44c0-9319-a6b69ff0dd55"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:12:04.915876 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:04.915795 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5336db60-bfa7-44c0-9319-a6b69ff0dd55-kube-api-access-5bwdq" (OuterVolumeSpecName: "kube-api-access-5bwdq") pod "5336db60-bfa7-44c0-9319-a6b69ff0dd55" (UID: "5336db60-bfa7-44c0-9319-a6b69ff0dd55"). InnerVolumeSpecName "kube-api-access-5bwdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:12:05.014860 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.014818 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:05.014860 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.014849 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bwdq\" (UniqueName: \"kubernetes.io/projected/5336db60-bfa7-44c0-9319-a6b69ff0dd55-kube-api-access-5bwdq\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:05.014860 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.014862 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5336db60-bfa7-44c0-9319-a6b69ff0dd55-console-oauth-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:05.635991 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.635952 2579 generic.go:356] "Generic (PLEG): container finished" podID="5336db60-bfa7-44c0-9319-a6b69ff0dd55" containerID="5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f" exitCode=0 Jul 28 17:12:05.636502 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.636013 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c76bdc7-kz8kj" event={"ID":"5336db60-bfa7-44c0-9319-a6b69ff0dd55","Type":"ContainerDied","Data":"5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f"} Jul 28 17:12:05.636502 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.636014 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c76bdc7-kz8kj" Jul 28 17:12:05.636502 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.636081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c76bdc7-kz8kj" event={"ID":"5336db60-bfa7-44c0-9319-a6b69ff0dd55","Type":"ContainerDied","Data":"bc4b3010a4061fcf32d9020d46874659ce335c7427bda73b34c3eeb2a7e712cd"} Jul 28 17:12:05.636502 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.636100 2579 scope.go:117] "RemoveContainer" containerID="5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f" Jul 28 17:12:05.644653 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.644627 2579 scope.go:117] "RemoveContainer" containerID="5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f" Jul 28 17:12:05.644960 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:12:05.644939 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f\": container with ID starting with 5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f not found: ID does not exist" containerID="5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f" Jul 28 17:12:05.645010 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.644970 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f"} err="failed to get container status \"5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f\": rpc error: code = NotFound desc = could not find container \"5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f\": container with ID starting with 5ebf5c894fea3bac1b2b4e5a0e6cd56f3ea492f521620bd83bd66db13a926e6f not found: ID does not exist" Jul 28 17:12:05.659411 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.659377 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c76bdc7-kz8kj"] Jul 28 17:12:05.664570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.664547 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c76bdc7-kz8kj"] Jul 28 17:12:05.684143 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:05.684094 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5336db60-bfa7-44c0-9319-a6b69ff0dd55" path="/var/lib/kubelet/pods/5336db60-bfa7-44c0-9319-a6b69ff0dd55/volumes" Jul 28 17:12:08.006743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:08.006709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:08.071071 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:08.071028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:25.786156 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:25.786119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:25.884751 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:25.884709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:26.453803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.453762 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns"] Jul 28 17:12:26.454043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.454031 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5336db60-bfa7-44c0-9319-a6b69ff0dd55" containerName="console" Jul 28 17:12:26.454086 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.454045 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5336db60-bfa7-44c0-9319-a6b69ff0dd55" containerName="console" Jul 28 17:12:26.454086 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.454056 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09d7394-2bea-4d64-bbe8-9de773d0c982" containerName="console" Jul 28 17:12:26.454086 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.454062 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09d7394-2bea-4d64-bbe8-9de773d0c982" containerName="console" Jul 28 17:12:26.454205 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.454103 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5336db60-bfa7-44c0-9319-a6b69ff0dd55" containerName="console" Jul 28 17:12:26.454205 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.454125 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09d7394-2bea-4d64-bbe8-9de773d0c982" containerName="console" Jul 28 17:12:26.458549 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.458528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.462298 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.462269 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 17:12:26.462501 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.462488 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cn7mb\"" Jul 28 17:12:26.463742 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.463717 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 17:12:26.471677 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.471647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns"] Jul 28 17:12:26.570070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.570028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.570264 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.570078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdlrg\" (UniqueName: \"kubernetes.io/projected/01c9f844-efcc-401f-a599-3092b216f675-kube-api-access-cdlrg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.570264 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.570152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.671238 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.671200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.671377 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.671248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdlrg\" (UniqueName: \"kubernetes.io/projected/01c9f844-efcc-401f-a599-3092b216f675-kube-api-access-cdlrg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.671377 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.671275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.671651 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.671636 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.671708 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.671635 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.681705 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.681678 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdlrg\" (UniqueName: \"kubernetes.io/projected/01c9f844-efcc-401f-a599-3092b216f675-kube-api-access-cdlrg\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.767307 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.767208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:26.900804 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:26.900662 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns"] Jul 28 17:12:26.903257 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:12:26.903225 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c9f844_efcc_401f_a599_3092b216f675.slice/crio-ce3ff0959cb0b0ace3778dee1e61e40d38837cca0178d7a154d4b2463b62f3c8 WatchSource:0}: Error finding container ce3ff0959cb0b0ace3778dee1e61e40d38837cca0178d7a154d4b2463b62f3c8: Status 404 returned error can't find the container with id ce3ff0959cb0b0ace3778dee1e61e40d38837cca0178d7a154d4b2463b62f3c8 Jul 28 17:12:27.712871 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:27.712832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" event={"ID":"01c9f844-efcc-401f-a599-3092b216f675","Type":"ContainerStarted","Data":"ce3ff0959cb0b0ace3778dee1e61e40d38837cca0178d7a154d4b2463b62f3c8"} Jul 28 17:12:34.741693 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:34.741653 2579 generic.go:356] "Generic (PLEG): container finished" podID="01c9f844-efcc-401f-a599-3092b216f675" containerID="e0e900d79badcedb0acb85cc09016501fd27e4acc4cd65e4699855bff0bdf448" exitCode=0 Jul 28 17:12:34.742068 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:34.741734 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" event={"ID":"01c9f844-efcc-401f-a599-3092b216f675","Type":"ContainerDied","Data":"e0e900d79badcedb0acb85cc09016501fd27e4acc4cd65e4699855bff0bdf448"} Jul 28 17:12:36.803153 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:36.803099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:40.766453 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:40.766413 2579 generic.go:356] "Generic (PLEG): container finished" podID="01c9f844-efcc-401f-a599-3092b216f675" containerID="73a319b55536649a3b726a1257c2e102fefe991b2edc09ad508d0e89786c96a7" exitCode=0 Jul 28 17:12:40.766954 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:40.766496 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" event={"ID":"01c9f844-efcc-401f-a599-3092b216f675","Type":"ContainerDied","Data":"73a319b55536649a3b726a1257c2e102fefe991b2edc09ad508d0e89786c96a7"} Jul 28 17:12:47.797179 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:47.797142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" event={"ID":"01c9f844-efcc-401f-a599-3092b216f675","Type":"ContainerStarted","Data":"9f5ce29e6fb6f35216edda1d12b6429298afc4a9f8d136151621d55845a2dcd8"} Jul 28 17:12:47.819131 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:47.819060 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" podStartSLOduration=2.194607198 podStartE2EDuration="21.819042401s" totalImagesPullingTime="19.624435203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:47.816799957 +0000 UTC m=+264.620488575" watchObservedRunningTime="2026-07-28 17:12:47.819042401 +0000 UTC m=+264.622730991" Jul 28 17:12:48.802554 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:48.802513 2579 generic.go:356] "Generic (PLEG): container finished" podID="01c9f844-efcc-401f-a599-3092b216f675" containerID="9f5ce29e6fb6f35216edda1d12b6429298afc4a9f8d136151621d55845a2dcd8" exitCode=0 Jul 28 17:12:48.802955 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:48.802589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" event={"ID":"01c9f844-efcc-401f-a599-3092b216f675","Type":"ContainerDied","Data":"9f5ce29e6fb6f35216edda1d12b6429298afc4a9f8d136151621d55845a2dcd8"} Jul 28 17:12:49.934101 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:49.934063 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:50.068682 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.068561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-bundle\") pod \"01c9f844-efcc-401f-a599-3092b216f675\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " Jul 28 17:12:50.068682 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.068643 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdlrg\" (UniqueName: \"kubernetes.io/projected/01c9f844-efcc-401f-a599-3092b216f675-kube-api-access-cdlrg\") pod \"01c9f844-efcc-401f-a599-3092b216f675\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " Jul 28 17:12:50.068894 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.068707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-util\") pod \"01c9f844-efcc-401f-a599-3092b216f675\" (UID: \"01c9f844-efcc-401f-a599-3092b216f675\") " Jul 28 17:12:50.069398 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.069364 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-bundle" (OuterVolumeSpecName: "bundle") pod "01c9f844-efcc-401f-a599-3092b216f675" (UID: "01c9f844-efcc-401f-a599-3092b216f675"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:12:50.071005 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.070970 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c9f844-efcc-401f-a599-3092b216f675-kube-api-access-cdlrg" (OuterVolumeSpecName: "kube-api-access-cdlrg") pod "01c9f844-efcc-401f-a599-3092b216f675" (UID: "01c9f844-efcc-401f-a599-3092b216f675"). InnerVolumeSpecName "kube-api-access-cdlrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:12:50.074045 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.074014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-util" (OuterVolumeSpecName: "util") pod "01c9f844-efcc-401f-a599-3092b216f675" (UID: "01c9f844-efcc-401f-a599-3092b216f675"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:12:50.170139 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.170064 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdlrg\" (UniqueName: \"kubernetes.io/projected/01c9f844-efcc-401f-a599-3092b216f675-kube-api-access-cdlrg\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:50.170139 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.170102 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-util\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:50.170139 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.170144 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/01c9f844-efcc-401f-a599-3092b216f675-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:12:50.811983 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.811958 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" Jul 28 17:12:50.812175 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.811950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kx4ns" event={"ID":"01c9f844-efcc-401f-a599-3092b216f675","Type":"ContainerDied","Data":"ce3ff0959cb0b0ace3778dee1e61e40d38837cca0178d7a154d4b2463b62f3c8"} Jul 28 17:12:50.812175 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:50.812079 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce3ff0959cb0b0ace3778dee1e61e40d38837cca0178d7a154d4b2463b62f3c8" Jul 28 17:12:52.802313 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:52.802270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:53.124428 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:53.124386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:53.571810 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:53.571707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:56.369180 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:56.369144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:56.475336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:56.475300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:58.695176 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:58.695091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:58.787845 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:58.787812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:59.024241 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.024137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:59.157818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.157771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:12:59.290768 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.290682 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p"] Jul 28 17:12:59.290953 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.290939 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c9f844-efcc-401f-a599-3092b216f675" containerName="pull" Jul 28 17:12:59.291001 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.290956 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c9f844-efcc-401f-a599-3092b216f675" containerName="pull" Jul 28 17:12:59.291001 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.290970 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c9f844-efcc-401f-a599-3092b216f675" containerName="util" Jul 28 17:12:59.291001 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.290976 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c9f844-efcc-401f-a599-3092b216f675" containerName="util" Jul 28 17:12:59.291001 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.290987 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c9f844-efcc-401f-a599-3092b216f675" containerName="extract" Jul 28 17:12:59.291001 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.290993 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c9f844-efcc-401f-a599-3092b216f675" containerName="extract" Jul 28 17:12:59.291176 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.291037 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c9f844-efcc-401f-a599-3092b216f675" containerName="extract" Jul 28 17:12:59.294407 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.294385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.340651 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.340606 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 17:12:59.340854 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.340680 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 17:12:59.340854 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.340684 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cn7mb\"" Jul 28 17:12:59.370773 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.370735 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p"] Jul 28 17:12:59.438981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.438936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.439194 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.439002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.439194 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.439066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7vz\" (UniqueName: \"kubernetes.io/projected/31d39181-bccc-4b9c-89cf-586cc7481cfa-kube-api-access-xz7vz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.540308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.540266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.540474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.540323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7vz\" (UniqueName: \"kubernetes.io/projected/31d39181-bccc-4b9c-89cf-586cc7481cfa-kube-api-access-xz7vz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.540474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.540350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.540680 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.540659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.540738 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.540715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.550562 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.550501 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7vz\" (UniqueName: \"kubernetes.io/projected/31d39181-bccc-4b9c-89cf-586cc7481cfa-kube-api-access-xz7vz\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.603338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.603282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:12:59.737843 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.737806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p"] Jul 28 17:12:59.741243 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:12:59.741206 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d39181_bccc_4b9c_89cf_586cc7481cfa.slice/crio-a6b6f239defcd132843b416cf0474853600d826903e9254f48db93f10b9bec57 WatchSource:0}: Error finding container a6b6f239defcd132843b416cf0474853600d826903e9254f48db93f10b9bec57: Status 404 returned error can't find the container with id a6b6f239defcd132843b416cf0474853600d826903e9254f48db93f10b9bec57 Jul 28 17:12:59.844578 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.844541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" event={"ID":"31d39181-bccc-4b9c-89cf-586cc7481cfa","Type":"ContainerStarted","Data":"8720d8b65cae923bebf5052523e1907ccf4d09ecaafbc9f004e49be75fc5c41d"} Jul 28 17:12:59.844721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:12:59.844597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" event={"ID":"31d39181-bccc-4b9c-89cf-586cc7481cfa","Type":"ContainerStarted","Data":"a6b6f239defcd132843b416cf0474853600d826903e9254f48db93f10b9bec57"} Jul 28 17:13:00.746615 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:00.746576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:00.849693 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:00.849658 2579 generic.go:356] "Generic (PLEG): container finished" podID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerID="8720d8b65cae923bebf5052523e1907ccf4d09ecaafbc9f004e49be75fc5c41d" exitCode=0 Jul 28 17:13:00.849898 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:00.849726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" event={"ID":"31d39181-bccc-4b9c-89cf-586cc7481cfa","Type":"ContainerDied","Data":"8720d8b65cae923bebf5052523e1907ccf4d09ecaafbc9f004e49be75fc5c41d"} Jul 28 17:13:01.012170 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.012131 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-59m6r"] Jul 28 17:13:01.016156 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.016129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.021652 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.021628 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 17:13:01.021837 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.021816 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 17:13:01.022063 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.022038 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-l6rwf\"" Jul 28 17:13:01.032064 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.032034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-59m6r"] Jul 28 17:13:01.153440 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.153345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d55804d2-21f0-4bb3-9c8f-c65930d7f801-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-59m6r\" (UID: \"d55804d2-21f0-4bb3-9c8f-c65930d7f801\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.153440 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.153393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66c42\" (UniqueName: \"kubernetes.io/projected/d55804d2-21f0-4bb3-9c8f-c65930d7f801-kube-api-access-66c42\") pod \"cert-manager-cainjector-559cdd8c87-59m6r\" (UID: \"d55804d2-21f0-4bb3-9c8f-c65930d7f801\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.242321 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.242290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:01.254317 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.254278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d55804d2-21f0-4bb3-9c8f-c65930d7f801-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-59m6r\" (UID: \"d55804d2-21f0-4bb3-9c8f-c65930d7f801\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.254463 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.254330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66c42\" (UniqueName: \"kubernetes.io/projected/d55804d2-21f0-4bb3-9c8f-c65930d7f801-kube-api-access-66c42\") pod \"cert-manager-cainjector-559cdd8c87-59m6r\" (UID: \"d55804d2-21f0-4bb3-9c8f-c65930d7f801\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.267880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.267848 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d55804d2-21f0-4bb3-9c8f-c65930d7f801-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-59m6r\" (UID: \"d55804d2-21f0-4bb3-9c8f-c65930d7f801\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.268026 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.268007 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66c42\" (UniqueName: \"kubernetes.io/projected/d55804d2-21f0-4bb3-9c8f-c65930d7f801-kube-api-access-66c42\") pod \"cert-manager-cainjector-559cdd8c87-59m6r\" (UID: \"d55804d2-21f0-4bb3-9c8f-c65930d7f801\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.325177 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.325140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" Jul 28 17:13:01.477137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.477087 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-59m6r"] Jul 28 17:13:01.480484 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:13:01.480453 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd55804d2_21f0_4bb3_9c8f_c65930d7f801.slice/crio-94e925f568227d883ada1263845bdb4389712e3042f43a6ae320a4778ab27279 WatchSource:0}: Error finding container 94e925f568227d883ada1263845bdb4389712e3042f43a6ae320a4778ab27279: Status 404 returned error can't find the container with id 94e925f568227d883ada1263845bdb4389712e3042f43a6ae320a4778ab27279 Jul 28 17:13:01.856712 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:01.856676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" event={"ID":"d55804d2-21f0-4bb3-9c8f-c65930d7f801","Type":"ContainerStarted","Data":"94e925f568227d883ada1263845bdb4389712e3042f43a6ae320a4778ab27279"} Jul 28 17:13:09.425605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:09.425565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:11.895161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:11.895124 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" event={"ID":"d55804d2-21f0-4bb3-9c8f-c65930d7f801","Type":"ContainerStarted","Data":"536c072b053dbec05818dd2abbe71cc76bd725cea5fbfba08b90a170a8b4d234"} Jul 28 17:13:11.917290 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:11.917235 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-59m6r" podStartSLOduration=2.2875743760000002 podStartE2EDuration="11.917218191s" totalImagesPullingTime="9.629643815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:11.915052441 +0000 UTC m=+288.718741024" watchObservedRunningTime="2026-07-28 17:13:11.917218191 +0000 UTC m=+288.720906780" Jul 28 17:13:11.970068 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:11.970029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:13.904634 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:13.904592 2579 generic.go:356] "Generic (PLEG): container finished" podID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerID="ec941806b77ec2b7b57adb19b5cda5fa1804fb59b62990b80c07c42ce55d8113" exitCode=0 Jul 28 17:13:13.905026 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:13.904673 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" event={"ID":"31d39181-bccc-4b9c-89cf-586cc7481cfa","Type":"ContainerDied","Data":"ec941806b77ec2b7b57adb19b5cda5fa1804fb59b62990b80c07c42ce55d8113"} Jul 28 17:13:14.910746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:14.910703 2579 generic.go:356] "Generic (PLEG): container finished" podID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerID="ed2247cc234f55d4b0a3f8331e170e69b1b1ac4393f81eb9ee81824b538f854f" exitCode=0 Jul 28 17:13:14.911256 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:14.910784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" event={"ID":"31d39181-bccc-4b9c-89cf-586cc7481cfa","Type":"ContainerDied","Data":"ed2247cc234f55d4b0a3f8331e170e69b1b1ac4393f81eb9ee81824b538f854f"} Jul 28 17:13:16.044430 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.044402 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:13:16.070244 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.070212 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-util\") pod \"31d39181-bccc-4b9c-89cf-586cc7481cfa\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " Jul 28 17:13:16.070244 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.070256 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-bundle\") pod \"31d39181-bccc-4b9c-89cf-586cc7481cfa\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " Jul 28 17:13:16.070718 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.070692 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-bundle" (OuterVolumeSpecName: "bundle") pod "31d39181-bccc-4b9c-89cf-586cc7481cfa" (UID: "31d39181-bccc-4b9c-89cf-586cc7481cfa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:13:16.075099 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.075068 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-util" (OuterVolumeSpecName: "util") pod "31d39181-bccc-4b9c-89cf-586cc7481cfa" (UID: "31d39181-bccc-4b9c-89cf-586cc7481cfa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:13:16.171064 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.170962 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz7vz\" (UniqueName: \"kubernetes.io/projected/31d39181-bccc-4b9c-89cf-586cc7481cfa-kube-api-access-xz7vz\") pod \"31d39181-bccc-4b9c-89cf-586cc7481cfa\" (UID: \"31d39181-bccc-4b9c-89cf-586cc7481cfa\") " Jul 28 17:13:16.171262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.171189 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-util\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:13:16.171262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.171208 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31d39181-bccc-4b9c-89cf-586cc7481cfa-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:13:16.173082 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.173058 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d39181-bccc-4b9c-89cf-586cc7481cfa-kube-api-access-xz7vz" (OuterVolumeSpecName: "kube-api-access-xz7vz") pod "31d39181-bccc-4b9c-89cf-586cc7481cfa" (UID: "31d39181-bccc-4b9c-89cf-586cc7481cfa"). InnerVolumeSpecName "kube-api-access-xz7vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:13:16.271548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.271496 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7vz\" (UniqueName: \"kubernetes.io/projected/31d39181-bccc-4b9c-89cf-586cc7481cfa-kube-api-access-xz7vz\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:13:16.919770 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.919728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" event={"ID":"31d39181-bccc-4b9c-89cf-586cc7481cfa","Type":"ContainerDied","Data":"a6b6f239defcd132843b416cf0474853600d826903e9254f48db93f10b9bec57"} Jul 28 17:13:16.919770 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.919760 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftbn5p" Jul 28 17:13:16.919989 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:16.919792 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b6f239defcd132843b416cf0474853600d826903e9254f48db93f10b9bec57" Jul 28 17:13:20.180976 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:20.180941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:20.289273 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:20.289233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:21.618093 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618050 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk"] Jul 28 17:13:21.618496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618313 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerName="util" Jul 28 17:13:21.618496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618325 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerName="util" Jul 28 17:13:21.618496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618333 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerName="pull" Jul 28 17:13:21.618496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618339 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerName="pull" Jul 28 17:13:21.618496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618354 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerName="extract" Jul 28 17:13:21.618496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618359 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerName="extract" Jul 28 17:13:21.618496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.618403 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d39181-bccc-4b9c-89cf-586cc7481cfa" containerName="extract" Jul 28 17:13:21.620254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.620237 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:21.623837 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.623810 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-7bnjr\"" Jul 28 17:13:21.624995 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.624970 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:13:21.625091 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.625069 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 17:13:21.640476 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.640446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk"] Jul 28 17:13:21.705796 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.705753 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21bdaf95-f0ad-4c98-9607-a570696f4c02-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w46nk\" (UID: \"21bdaf95-f0ad-4c98-9607-a570696f4c02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:21.705796 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.705791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4d8\" (UniqueName: \"kubernetes.io/projected/21bdaf95-f0ad-4c98-9607-a570696f4c02-kube-api-access-mv4d8\") pod \"openshift-lws-operator-fd8ccff4c-w46nk\" (UID: \"21bdaf95-f0ad-4c98-9607-a570696f4c02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:21.806267 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.806220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21bdaf95-f0ad-4c98-9607-a570696f4c02-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w46nk\" (UID: \"21bdaf95-f0ad-4c98-9607-a570696f4c02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:21.806267 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.806270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4d8\" (UniqueName: \"kubernetes.io/projected/21bdaf95-f0ad-4c98-9607-a570696f4c02-kube-api-access-mv4d8\") pod \"openshift-lws-operator-fd8ccff4c-w46nk\" (UID: \"21bdaf95-f0ad-4c98-9607-a570696f4c02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:21.806632 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.806607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/21bdaf95-f0ad-4c98-9607-a570696f4c02-tmp\") pod \"openshift-lws-operator-fd8ccff4c-w46nk\" (UID: \"21bdaf95-f0ad-4c98-9607-a570696f4c02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:21.816999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.816970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4d8\" (UniqueName: \"kubernetes.io/projected/21bdaf95-f0ad-4c98-9607-a570696f4c02-kube-api-access-mv4d8\") pod \"openshift-lws-operator-fd8ccff4c-w46nk\" (UID: \"21bdaf95-f0ad-4c98-9607-a570696f4c02\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:21.929646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:21.929550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" Jul 28 17:13:22.094516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:22.094489 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk"] Jul 28 17:13:22.097550 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:13:22.097522 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21bdaf95_f0ad_4c98_9607_a570696f4c02.slice/crio-9ce3c4940ef4efaf4b24ccd574fc7b2e1cae6ce5a0f4b7ae7bed6ba784cca667 WatchSource:0}: Error finding container 9ce3c4940ef4efaf4b24ccd574fc7b2e1cae6ce5a0f4b7ae7bed6ba784cca667: Status 404 returned error can't find the container with id 9ce3c4940ef4efaf4b24ccd574fc7b2e1cae6ce5a0f4b7ae7bed6ba784cca667 Jul 28 17:13:22.943632 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:22.943590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" event={"ID":"21bdaf95-f0ad-4c98-9607-a570696f4c02","Type":"ContainerStarted","Data":"9ce3c4940ef4efaf4b24ccd574fc7b2e1cae6ce5a0f4b7ae7bed6ba784cca667"} Jul 28 17:13:23.952755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:23.952725 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:13:23.954049 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:23.952730 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:13:23.956758 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:23.956734 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 17:13:24.952687 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:24.952648 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" event={"ID":"21bdaf95-f0ad-4c98-9607-a570696f4c02","Type":"ContainerStarted","Data":"923de4e78994b9ecb133f81451e91f6394dc04a8ffca04bf894a05ee02fb7b90"} Jul 28 17:13:24.971955 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:24.971901 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-w46nk" podStartSLOduration=2.106998717 podStartE2EDuration="3.971882722s" totalImagesPullingTime="1.864884005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:24.969794585 +0000 UTC m=+301.773483175" watchObservedRunningTime="2026-07-28 17:13:24.971882722 +0000 UTC m=+301.775571312" Jul 28 17:13:25.840182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:25.840136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:27.122922 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.122881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:27.858330 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.858287 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph"] Jul 28 17:13:27.860650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.860624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:27.864500 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.864475 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cn7mb\"" Jul 28 17:13:27.864638 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.864571 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 17:13:27.868183 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.868161 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 17:13:27.893234 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.893201 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph"] Jul 28 17:13:27.953952 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.953910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjvsn\" (UniqueName: \"kubernetes.io/projected/53dcc3f4-c1b3-451a-9211-10b928341868-kube-api-access-pjvsn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:27.954144 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.953961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:27.954144 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.954002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:27.992324 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:27.992286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:28.054961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.054921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:28.055172 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.054985 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjvsn\" (UniqueName: \"kubernetes.io/projected/53dcc3f4-c1b3-451a-9211-10b928341868-kube-api-access-pjvsn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:28.055172 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.055024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:28.055375 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.055340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:28.055418 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.055388 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:28.072973 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.072937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjvsn\" (UniqueName: \"kubernetes.io/projected/53dcc3f4-c1b3-451a-9211-10b928341868-kube-api-access-pjvsn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:28.170374 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.170277 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:28.321402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.321367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph"] Jul 28 17:13:28.324289 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:13:28.324258 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53dcc3f4_c1b3_451a_9211_10b928341868.slice/crio-87de00531b20b55211414730838410d8178a693325bd0c848a4401198c018413 WatchSource:0}: Error finding container 87de00531b20b55211414730838410d8178a693325bd0c848a4401198c018413: Status 404 returned error can't find the container with id 87de00531b20b55211414730838410d8178a693325bd0c848a4401198c018413 Jul 28 17:13:28.969378 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.969345 2579 generic.go:356] "Generic (PLEG): container finished" podID="53dcc3f4-c1b3-451a-9211-10b928341868" containerID="e497aca06fe1b3650d688a161e804aa16ebd0fd3f3c42bc4e036e4918fa50e2c" exitCode=0 Jul 28 17:13:28.969533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.969390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" event={"ID":"53dcc3f4-c1b3-451a-9211-10b928341868","Type":"ContainerDied","Data":"e497aca06fe1b3650d688a161e804aa16ebd0fd3f3c42bc4e036e4918fa50e2c"} Jul 28 17:13:28.969533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.969424 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" event={"ID":"53dcc3f4-c1b3-451a-9211-10b928341868","Type":"ContainerStarted","Data":"87de00531b20b55211414730838410d8178a693325bd0c848a4401198c018413"} Jul 28 17:13:28.969882 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:28.969867 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:13:29.975048 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:29.975005 2579 generic.go:356] "Generic (PLEG): container finished" podID="53dcc3f4-c1b3-451a-9211-10b928341868" containerID="bc22e7e1c6c7e0704ced07d10240bf7b91379cf9c9a9cbfebab74cc1e8686e35" exitCode=0 Jul 28 17:13:29.975478 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:29.975088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" event={"ID":"53dcc3f4-c1b3-451a-9211-10b928341868","Type":"ContainerDied","Data":"bc22e7e1c6c7e0704ced07d10240bf7b91379cf9c9a9cbfebab74cc1e8686e35"} Jul 28 17:13:30.981229 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:30.981184 2579 generic.go:356] "Generic (PLEG): container finished" podID="53dcc3f4-c1b3-451a-9211-10b928341868" containerID="4750a2753ce92c81da5029d5e44373b55a02bf42bd4a04ff83fd0c1106fe5274" exitCode=0 Jul 28 17:13:30.981670 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:30.981273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" event={"ID":"53dcc3f4-c1b3-451a-9211-10b928341868","Type":"ContainerDied","Data":"4750a2753ce92c81da5029d5e44373b55a02bf42bd4a04ff83fd0c1106fe5274"} Jul 28 17:13:32.116736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.116702 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:32.168647 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.168608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:32.189218 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.189186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-util\") pod \"53dcc3f4-c1b3-451a-9211-10b928341868\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " Jul 28 17:13:32.189424 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.189237 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjvsn\" (UniqueName: \"kubernetes.io/projected/53dcc3f4-c1b3-451a-9211-10b928341868-kube-api-access-pjvsn\") pod \"53dcc3f4-c1b3-451a-9211-10b928341868\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " Jul 28 17:13:32.189424 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.189275 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-bundle\") pod \"53dcc3f4-c1b3-451a-9211-10b928341868\" (UID: \"53dcc3f4-c1b3-451a-9211-10b928341868\") " Jul 28 17:13:32.190006 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.189977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-bundle" (OuterVolumeSpecName: "bundle") pod "53dcc3f4-c1b3-451a-9211-10b928341868" (UID: "53dcc3f4-c1b3-451a-9211-10b928341868"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:13:32.191422 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.191389 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53dcc3f4-c1b3-451a-9211-10b928341868-kube-api-access-pjvsn" (OuterVolumeSpecName: "kube-api-access-pjvsn") pod "53dcc3f4-c1b3-451a-9211-10b928341868" (UID: "53dcc3f4-c1b3-451a-9211-10b928341868"). InnerVolumeSpecName "kube-api-access-pjvsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:13:32.194765 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.194740 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-util" (OuterVolumeSpecName: "util") pod "53dcc3f4-c1b3-451a-9211-10b928341868" (UID: "53dcc3f4-c1b3-451a-9211-10b928341868"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:13:32.246761 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.246674 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448"] Jul 28 17:13:32.246940 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.246928 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53dcc3f4-c1b3-451a-9211-10b928341868" containerName="pull" Jul 28 17:13:32.246990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.246941 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="53dcc3f4-c1b3-451a-9211-10b928341868" containerName="pull" Jul 28 17:13:32.246990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.246955 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53dcc3f4-c1b3-451a-9211-10b928341868" containerName="util" Jul 28 17:13:32.246990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.246961 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="53dcc3f4-c1b3-451a-9211-10b928341868" containerName="util" Jul 28 17:13:32.246990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.246969 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53dcc3f4-c1b3-451a-9211-10b928341868" containerName="extract" Jul 28 17:13:32.246990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.246974 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="53dcc3f4-c1b3-451a-9211-10b928341868" containerName="extract" Jul 28 17:13:32.247191 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.247023 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="53dcc3f4-c1b3-451a-9211-10b928341868" containerName="extract" Jul 28 17:13:32.249436 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.249417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.253837 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.253813 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 17:13:32.253961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.253815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-njktv\"" Jul 28 17:13:32.265343 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.265316 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 17:13:32.267237 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.267208 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 17:13:32.268560 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.268539 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448"] Jul 28 17:13:32.271650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.271620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:32.290603 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.290568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2xmj\" (UniqueName: \"kubernetes.io/projected/946fa23d-66f5-4ec5-b90e-eedb5a21f701-kube-api-access-h2xmj\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.290603 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.290607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946fa23d-66f5-4ec5-b90e-eedb5a21f701-cert\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.290816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.290647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/946fa23d-66f5-4ec5-b90e-eedb5a21f701-manager-config\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.290816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.290665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/946fa23d-66f5-4ec5-b90e-eedb5a21f701-metrics-cert\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.290816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.290702 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:13:32.290816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.290714 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53dcc3f4-c1b3-451a-9211-10b928341868-util\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:13:32.290816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.290723 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjvsn\" (UniqueName: \"kubernetes.io/projected/53dcc3f4-c1b3-451a-9211-10b928341868-kube-api-access-pjvsn\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:13:32.391930 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.391890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2xmj\" (UniqueName: \"kubernetes.io/projected/946fa23d-66f5-4ec5-b90e-eedb5a21f701-kube-api-access-h2xmj\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.391930 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.391932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946fa23d-66f5-4ec5-b90e-eedb5a21f701-cert\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.392262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.391964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/946fa23d-66f5-4ec5-b90e-eedb5a21f701-manager-config\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.392262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.391980 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/946fa23d-66f5-4ec5-b90e-eedb5a21f701-metrics-cert\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.392771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.392741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/946fa23d-66f5-4ec5-b90e-eedb5a21f701-manager-config\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.394520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.394487 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/946fa23d-66f5-4ec5-b90e-eedb5a21f701-metrics-cert\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.394621 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.394544 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/946fa23d-66f5-4ec5-b90e-eedb5a21f701-cert\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.402709 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.402676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2xmj\" (UniqueName: \"kubernetes.io/projected/946fa23d-66f5-4ec5-b90e-eedb5a21f701-kube-api-access-h2xmj\") pod \"lws-controller-manager-5cdc965c4d-lf448\" (UID: \"946fa23d-66f5-4ec5-b90e-eedb5a21f701\") " pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.558478 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.558359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:32.703532 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.703497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448"] Jul 28 17:13:32.706917 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:13:32.706892 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946fa23d_66f5_4ec5_b90e_eedb5a21f701.slice/crio-42c9a04e435e6789e799c6036629f926c1e8e65df540fb5437585b424c3028be WatchSource:0}: Error finding container 42c9a04e435e6789e799c6036629f926c1e8e65df540fb5437585b424c3028be: Status 404 returned error can't find the container with id 42c9a04e435e6789e799c6036629f926c1e8e65df540fb5437585b424c3028be Jul 28 17:13:32.989987 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.989948 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" event={"ID":"946fa23d-66f5-4ec5-b90e-eedb5a21f701","Type":"ContainerStarted","Data":"42c9a04e435e6789e799c6036629f926c1e8e65df540fb5437585b424c3028be"} Jul 28 17:13:32.991722 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.991688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" event={"ID":"53dcc3f4-c1b3-451a-9211-10b928341868","Type":"ContainerDied","Data":"87de00531b20b55211414730838410d8178a693325bd0c848a4401198c018413"} Jul 28 17:13:32.991722 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.991715 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7xrph" Jul 28 17:13:32.991887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:32.991730 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87de00531b20b55211414730838410d8178a693325bd0c848a4401198c018413" Jul 28 17:13:35.000542 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:35.000501 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" event={"ID":"946fa23d-66f5-4ec5-b90e-eedb5a21f701","Type":"ContainerStarted","Data":"cd78de6dd3e22e41594a72cb083893af8da2f56c1d5501e97987a244e6707758"} Jul 28 17:13:35.001151 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:35.000646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:35.021729 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:35.021671 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" podStartSLOduration=1.43900093 podStartE2EDuration="3.021650064s" totalImagesPullingTime="1.582649134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:35.020474405 +0000 UTC m=+311.824162994" watchObservedRunningTime="2026-07-28 17:13:35.021650064 +0000 UTC m=+311.825338653" Jul 28 17:13:43.112248 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:43.112202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:43.224034 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:43.223996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:43.734155 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:43.734085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.006733 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.006637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.158694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.158659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.211053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.211014 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5"] Jul 28 17:13:44.213340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.213323 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.217345 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.217321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 17:13:44.217506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.217424 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 17:13:44.217575 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.217543 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 17:13:44.217688 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.217671 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 17:13:44.217744 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.217722 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m9txd\"" Jul 28 17:13:44.242802 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.242768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5"] Jul 28 17:13:44.279862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.279768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddg8\" (UniqueName: \"kubernetes.io/projected/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-kube-api-access-nddg8\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.279862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.279809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.279862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.279829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.280175 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.279942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-tmp\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.280175 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.280004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-manifests\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.381145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.381089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddg8\" (UniqueName: \"kubernetes.io/projected/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-kube-api-access-nddg8\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.381145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.381149 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.381411 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.381182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.381411 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.381224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-tmp\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.381411 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.381273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-manifests\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.381738 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.381706 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-manifests\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.383527 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.383491 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-tmp\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.383696 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.383675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.383748 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.383729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-webhook-cert\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.394916 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.394885 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddg8\" (UniqueName: \"kubernetes.io/projected/a50cf5e2-75e3-4927-8b16-b49bd2d57b59-kube-api-access-nddg8\") pod \"opendatahub-operator-controller-manager-d8c7b7dcd-q26l5\" (UID: \"a50cf5e2-75e3-4927-8b16-b49bd2d57b59\") " pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.458042 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.457999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.523325 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.523283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:44.559011 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.558967 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.643872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.643830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.672382 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.672357 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5"] Jul 28 17:13:44.674896 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:13:44.674867 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50cf5e2_75e3_4927_8b16_b49bd2d57b59.slice/crio-449e0946e82977b00019a0583ccbdc1751cf55d29d50ff9ea980c6e91a620272 WatchSource:0}: Error finding container 449e0946e82977b00019a0583ccbdc1751cf55d29d50ff9ea980c6e91a620272: Status 404 returned error can't find the container with id 449e0946e82977b00019a0583ccbdc1751cf55d29d50ff9ea980c6e91a620272 Jul 28 17:13:44.781797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.781760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.875268 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.875224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:44.984740 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:44.984707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:45.040912 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:45.040879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" event={"ID":"a50cf5e2-75e3-4927-8b16-b49bd2d57b59","Type":"ContainerStarted","Data":"449e0946e82977b00019a0583ccbdc1751cf55d29d50ff9ea980c6e91a620272"} Jul 28 17:13:45.076477 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:45.076438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:13:46.006571 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:46.006536 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5cdc965c4d-lf448" Jul 28 17:13:48.055728 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:48.055689 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" event={"ID":"a50cf5e2-75e3-4927-8b16-b49bd2d57b59","Type":"ContainerStarted","Data":"b4b864aeaab9159841cb86a50041fd29d4642a1fbee4483749e20455624f2449"} Jul 28 17:13:49.061434 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:49.061399 2579 generic.go:356] "Generic (PLEG): container finished" podID="a50cf5e2-75e3-4927-8b16-b49bd2d57b59" containerID="b4b864aeaab9159841cb86a50041fd29d4642a1fbee4483749e20455624f2449" exitCode=0 Jul 28 17:13:49.061875 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:49.061480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" event={"ID":"a50cf5e2-75e3-4927-8b16-b49bd2d57b59","Type":"ContainerDied","Data":"b4b864aeaab9159841cb86a50041fd29d4642a1fbee4483749e20455624f2449"} Jul 28 17:13:50.068513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:50.068478 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" event={"ID":"a50cf5e2-75e3-4927-8b16-b49bd2d57b59","Type":"ContainerStarted","Data":"f50dd4e0cab3de88ba6e47d0ef4255fe05b1eef454b2f0a7be27ea3ee82357e8"} Jul 28 17:13:50.068952 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:50.068612 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:13:50.096092 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:13:50.096038 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" podStartSLOduration=1.470269675 podStartE2EDuration="6.09601924s" totalImagesPullingTime="3.568628128s" totalInitContainerRuntime="1.057121437s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:13:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:13:50.095345223 +0000 UTC m=+326.899033836" watchObservedRunningTime="2026-07-28 17:13:50.09601924 +0000 UTC m=+326.899707830" Jul 28 17:14:01.074689 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:01.074658 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d8c7b7dcd-q26l5" Jul 28 17:14:03.430997 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:03.430954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:03.564222 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:03.564183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:04.257785 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:04.257726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:04.421153 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:04.421091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:04.726853 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:04.726817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:04.828819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:04.828778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:04.939841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:04.939775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:05.038316 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:05.038201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:05.763912 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:05.763864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:06.088407 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:06.088373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:06.191310 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:06.191268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:06.334421 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:06.334386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:07.798457 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:07.798419 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:11.640254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:11.640223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:11.876812 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:11.876744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:12.660481 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.660446 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m"] Jul 28 17:14:12.669964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.669930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.676922 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.676901 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 28 17:14:12.677489 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.677470 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 17:14:12.677569 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.677478 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 28 17:14:12.678648 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.678632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 28 17:14:12.678710 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.678636 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7dpg4\"" Jul 28 17:14:12.678885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.678873 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 17:14:12.679410 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.679398 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 17:14:12.694870 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.694840 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m"] Jul 28 17:14:12.815397 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.815359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b8ff\" (UniqueName: \"kubernetes.io/projected/0185fa99-d0d2-44c2-8747-db80d71a5ab2-kube-api-access-8b8ff\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.815397 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.815400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.815631 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.815427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.815631 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.815500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.815631 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.815518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.815631 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.815547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185fa99-d0d2-44c2-8747-db80d71a5ab2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.815631 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.815599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.916279 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.916181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.916279 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.916240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b8ff\" (UniqueName: \"kubernetes.io/projected/0185fa99-d0d2-44c2-8747-db80d71a5ab2-kube-api-access-8b8ff\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.916514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.916346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.916514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.916390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.916514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.916437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.916514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.916463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.916514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.916508 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185fa99-d0d2-44c2-8747-db80d71a5ab2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.917340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.917308 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.919182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.919132 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0185fa99-d0d2-44c2-8747-db80d71a5ab2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.919559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.919528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.919918 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.919896 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.920023 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.919933 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.952311 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.952283 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0185fa99-d0d2-44c2-8747-db80d71a5ab2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.952619 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.952597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b8ff\" (UniqueName: \"kubernetes.io/projected/0185fa99-d0d2-44c2-8747-db80d71a5ab2-kube-api-access-8b8ff\") pod \"istiod-openshift-gateway-5677ccb49b-lqs6m\" (UID: \"0185fa99-d0d2-44c2-8747-db80d71a5ab2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:12.967370 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.967322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:12.979678 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:12.979654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:13.157283 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:13.157243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:13.201503 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:14:13.201470 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0185fa99_d0d2_44c2_8747_db80d71a5ab2.slice/crio-1c3a0b70fbcf62bba5eafe0dab8fc17002ee2e1992f7c8c35315aa0c52c09559 WatchSource:0}: Error finding container 1c3a0b70fbcf62bba5eafe0dab8fc17002ee2e1992f7c8c35315aa0c52c09559: Status 404 returned error can't find the container with id 1c3a0b70fbcf62bba5eafe0dab8fc17002ee2e1992f7c8c35315aa0c52c09559 Jul 28 17:14:13.203185 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:13.203163 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m"] Jul 28 17:14:13.349850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:13.349799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:14.175608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:14.175563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" event={"ID":"0185fa99-d0d2-44c2-8747-db80d71a5ab2","Type":"ContainerStarted","Data":"1c3a0b70fbcf62bba5eafe0dab8fc17002ee2e1992f7c8c35315aa0c52c09559"} Jul 28 17:14:15.737670 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:15.737616 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:14:15.737956 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:15.737710 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:14:16.188251 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:16.188209 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" event={"ID":"0185fa99-d0d2-44c2-8747-db80d71a5ab2","Type":"ContainerStarted","Data":"8ebfcdca1538279495291bfdb48e7dd4e5ba7d3706f415a1d3cad75ba1b75b9f"} Jul 28 17:14:16.188442 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:16.188422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:16.190182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:16.190152 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-lqs6m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 17:14:16.190293 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:16.190204 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" podUID="0185fa99-d0d2-44c2-8747-db80d71a5ab2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:14:16.236596 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:16.236537 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" podStartSLOduration=1.7028160190000001 podStartE2EDuration="4.236520575s" totalImagesPullingTime="2.533704556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:14:16.236191435 +0000 UTC m=+353.039880027" watchObservedRunningTime="2026-07-28 17:14:16.236520575 +0000 UTC m=+353.040209166" Jul 28 17:14:17.193864 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:17.193827 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-lqs6m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 17:14:17.194445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:17.193888 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" podUID="0185fa99-d0d2-44c2-8747-db80d71a5ab2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:14:18.180028 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:18.179994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:18.368410 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:18.368369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:18.515919 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:18.515826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:18.619354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:18.619319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:19.274188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:19.274146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:19.685819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:19.685782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:19.889023 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:19.888986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:19.985335 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:19.985229 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:20.194845 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:20.194814 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-lqs6m" Jul 28 17:14:31.992050 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:31.992011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:40.955102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:40.955062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:49.299123 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:49.299071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:56.392939 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:56.392894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:14:56.495455 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:14:56.495420 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:04.921525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:04.921479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:05.021298 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:05.021259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:05.102308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:05.102266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:20.450370 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.450333 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8"] Jul 28 17:15:20.453575 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.453559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.456610 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.456591 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 17:15:20.457647 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.457617 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 17:15:20.457766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.457672 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m4mtg\"" Jul 28 17:15:20.464269 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.464247 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8"] Jul 28 17:15:20.536392 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.536355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.536568 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.536401 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgpr\" (UniqueName: \"kubernetes.io/projected/7c8ac905-db82-43da-812c-9bbac86ee418-kube-api-access-6wgpr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.536568 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.536424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.637161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.637082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.637320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.637187 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgpr\" (UniqueName: \"kubernetes.io/projected/7c8ac905-db82-43da-812c-9bbac86ee418-kube-api-access-6wgpr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.637320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.637214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.637488 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.637467 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.637543 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.637524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.647861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.647835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgpr\" (UniqueName: \"kubernetes.io/projected/7c8ac905-db82-43da-812c-9bbac86ee418-kube-api-access-6wgpr\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.762828 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.762742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:20.892898 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:20.892870 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8"] Jul 28 17:15:20.894998 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:15:20.894964 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8ac905_db82_43da_812c_9bbac86ee418.slice/crio-787c07e6d86551137cca1b89b8ec61a81c5af802b6ee991db5e2c5dac4c2c307 WatchSource:0}: Error finding container 787c07e6d86551137cca1b89b8ec61a81c5af802b6ee991db5e2c5dac4c2c307: Status 404 returned error can't find the container with id 787c07e6d86551137cca1b89b8ec61a81c5af802b6ee991db5e2c5dac4c2c307 Jul 28 17:15:21.061888 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.061814 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp"] Jul 28 17:15:21.065039 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.065020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.080992 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.080967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp"] Jul 28 17:15:21.141830 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.141796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.142008 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.141848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.142008 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.141870 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bqw9\" (UniqueName: \"kubernetes.io/projected/6a8fecb2-2a21-40ef-aa06-9b4c84535295-kube-api-access-5bqw9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.242743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.242686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.242743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.242759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.243076 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.242784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bqw9\" (UniqueName: \"kubernetes.io/projected/6a8fecb2-2a21-40ef-aa06-9b4c84535295-kube-api-access-5bqw9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.243253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.243223 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.243326 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.243275 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.253509 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.253483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bqw9\" (UniqueName: \"kubernetes.io/projected/6a8fecb2-2a21-40ef-aa06-9b4c84535295-kube-api-access-5bqw9\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.373592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.373539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:21.470219 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.470187 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c8ac905-db82-43da-812c-9bbac86ee418" containerID="666237317af8ac6d8a856cd5a3348ba36bd6054204b04a0b0b1cce34646987e9" exitCode=0 Jul 28 17:15:21.470687 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.470273 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" event={"ID":"7c8ac905-db82-43da-812c-9bbac86ee418","Type":"ContainerDied","Data":"666237317af8ac6d8a856cd5a3348ba36bd6054204b04a0b0b1cce34646987e9"} Jul 28 17:15:21.470687 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.470341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" event={"ID":"7c8ac905-db82-43da-812c-9bbac86ee418","Type":"ContainerStarted","Data":"787c07e6d86551137cca1b89b8ec61a81c5af802b6ee991db5e2c5dac4c2c307"} Jul 28 17:15:21.479293 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.479260 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd"] Jul 28 17:15:21.482803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.482777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.494691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.494666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd"] Jul 28 17:15:21.524202 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.524174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp"] Jul 28 17:15:21.525905 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:15:21.525879 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8fecb2_2a21_40ef_aa06_9b4c84535295.slice/crio-5cbed05198dea9ef646515a1f06fb14ae1805064e8194afeafbf85be30b7ec89 WatchSource:0}: Error finding container 5cbed05198dea9ef646515a1f06fb14ae1805064e8194afeafbf85be30b7ec89: Status 404 returned error can't find the container with id 5cbed05198dea9ef646515a1f06fb14ae1805064e8194afeafbf85be30b7ec89 Jul 28 17:15:21.544836 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.544802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.544930 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.544845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcq2\" (UniqueName: \"kubernetes.io/projected/fc612252-cc0f-4016-9b05-33cbf44cfc97-kube-api-access-nqcq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.544987 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.544967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.646243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.646146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.646243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.646190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcq2\" (UniqueName: \"kubernetes.io/projected/fc612252-cc0f-4016-9b05-33cbf44cfc97-kube-api-access-nqcq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.646425 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.646279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.646635 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.646612 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.646671 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.646615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.660220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.660197 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcq2\" (UniqueName: \"kubernetes.io/projected/fc612252-cc0f-4016-9b05-33cbf44cfc97-kube-api-access-nqcq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.793092 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.793056 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:21.858329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.858283 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg"] Jul 28 17:15:21.862331 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.862294 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:21.879033 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.877606 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg"] Jul 28 17:15:21.940665 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.940582 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd"] Jul 28 17:15:21.943687 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:15:21.943660 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc612252_cc0f_4016_9b05_33cbf44cfc97.slice/crio-20e02480e01816f6d722362556f12160828786f0b7fd9d2d18b1ae9eca9becc1 WatchSource:0}: Error finding container 20e02480e01816f6d722362556f12160828786f0b7fd9d2d18b1ae9eca9becc1: Status 404 returned error can't find the container with id 20e02480e01816f6d722362556f12160828786f0b7fd9d2d18b1ae9eca9becc1 Jul 28 17:15:21.948775 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.948746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:21.948887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.948815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:21.948956 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:21.948928 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcrgd\" (UniqueName: \"kubernetes.io/projected/e62a862c-3584-4f5c-a084-0a29427876d7-kube-api-access-mcrgd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.050344 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.050311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.050489 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.050412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcrgd\" (UniqueName: \"kubernetes.io/projected/e62a862c-3584-4f5c-a084-0a29427876d7-kube-api-access-mcrgd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.050489 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.050470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.050681 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.050657 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.050749 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.050710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.060888 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.060860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcrgd\" (UniqueName: \"kubernetes.io/projected/e62a862c-3584-4f5c-a084-0a29427876d7-kube-api-access-mcrgd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.185667 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.185641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:22.341416 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.341391 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg"] Jul 28 17:15:22.343545 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:15:22.343523 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62a862c_3584_4f5c_a084_0a29427876d7.slice/crio-7770862e994e5eefb59f3c6893f97ae8f0e2d251c66620d38fda70f1eb7d94ce WatchSource:0}: Error finding container 7770862e994e5eefb59f3c6893f97ae8f0e2d251c66620d38fda70f1eb7d94ce: Status 404 returned error can't find the container with id 7770862e994e5eefb59f3c6893f97ae8f0e2d251c66620d38fda70f1eb7d94ce Jul 28 17:15:22.477633 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.477539 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c8ac905-db82-43da-812c-9bbac86ee418" containerID="30d95661ec87ae0cb07054f5e33300227e7d0cbb856f44e24b4682341cd2aab5" exitCode=0 Jul 28 17:15:22.478086 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.477624 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" event={"ID":"7c8ac905-db82-43da-812c-9bbac86ee418","Type":"ContainerDied","Data":"30d95661ec87ae0cb07054f5e33300227e7d0cbb856f44e24b4682341cd2aab5"} Jul 28 17:15:22.479217 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.479098 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerID="d8644ea13d5ed143b0ad19f5f4703b71159d82b1545f74a19a92d8b02151b432" exitCode=0 Jul 28 17:15:22.479217 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.479138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" event={"ID":"fc612252-cc0f-4016-9b05-33cbf44cfc97","Type":"ContainerDied","Data":"d8644ea13d5ed143b0ad19f5f4703b71159d82b1545f74a19a92d8b02151b432"} Jul 28 17:15:22.479217 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.479184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" event={"ID":"fc612252-cc0f-4016-9b05-33cbf44cfc97","Type":"ContainerStarted","Data":"20e02480e01816f6d722362556f12160828786f0b7fd9d2d18b1ae9eca9becc1"} Jul 28 17:15:22.480809 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.480791 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerID="ccee80041cfe9a95b0166b518544dcc3f00111cc272fe3683a11c5d4e4052b09" exitCode=0 Jul 28 17:15:22.480928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.480878 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" event={"ID":"6a8fecb2-2a21-40ef-aa06-9b4c84535295","Type":"ContainerDied","Data":"ccee80041cfe9a95b0166b518544dcc3f00111cc272fe3683a11c5d4e4052b09"} Jul 28 17:15:22.480998 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.480948 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" event={"ID":"6a8fecb2-2a21-40ef-aa06-9b4c84535295","Type":"ContainerStarted","Data":"5cbed05198dea9ef646515a1f06fb14ae1805064e8194afeafbf85be30b7ec89"} Jul 28 17:15:22.482532 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.482510 2579 generic.go:356] "Generic (PLEG): container finished" podID="e62a862c-3584-4f5c-a084-0a29427876d7" containerID="00908929795f1c714fffcd53e16f2fcb2cc3836d8dd170a610f19af070e163ad" exitCode=0 Jul 28 17:15:22.482633 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.482542 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" event={"ID":"e62a862c-3584-4f5c-a084-0a29427876d7","Type":"ContainerDied","Data":"00908929795f1c714fffcd53e16f2fcb2cc3836d8dd170a610f19af070e163ad"} Jul 28 17:15:22.482633 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:22.482573 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" event={"ID":"e62a862c-3584-4f5c-a084-0a29427876d7","Type":"ContainerStarted","Data":"7770862e994e5eefb59f3c6893f97ae8f0e2d251c66620d38fda70f1eb7d94ce"} Jul 28 17:15:23.488841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:23.488798 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerID="98df3fa397f0a682a1fa986e1378872ceff3aa0aaf43c31550f1ba0939962959" exitCode=0 Jul 28 17:15:23.489312 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:23.488879 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" event={"ID":"fc612252-cc0f-4016-9b05-33cbf44cfc97","Type":"ContainerDied","Data":"98df3fa397f0a682a1fa986e1378872ceff3aa0aaf43c31550f1ba0939962959"} Jul 28 17:15:23.490961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:23.490939 2579 generic.go:356] "Generic (PLEG): container finished" podID="7c8ac905-db82-43da-812c-9bbac86ee418" containerID="53b8682238ad37d6f6201895fcac7f04bb604d6773c0040d8ef3b0827401db7d" exitCode=0 Jul 28 17:15:23.491057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:23.490984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" event={"ID":"7c8ac905-db82-43da-812c-9bbac86ee418","Type":"ContainerDied","Data":"53b8682238ad37d6f6201895fcac7f04bb604d6773c0040d8ef3b0827401db7d"} Jul 28 17:15:24.498150 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.498090 2579 generic.go:356] "Generic (PLEG): container finished" podID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerID="527404c306698d07a717985e8f87316dc91b7c3a60002fef44a309364c0062e4" exitCode=0 Jul 28 17:15:24.498563 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.498173 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" event={"ID":"fc612252-cc0f-4016-9b05-33cbf44cfc97","Type":"ContainerDied","Data":"527404c306698d07a717985e8f87316dc91b7c3a60002fef44a309364c0062e4"} Jul 28 17:15:24.499973 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.499950 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerID="019282b362875b23fb345d679a176a87bbd9464c087e1f31b08cfd7bb5130ce8" exitCode=0 Jul 28 17:15:24.500097 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.500017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" event={"ID":"6a8fecb2-2a21-40ef-aa06-9b4c84535295","Type":"ContainerDied","Data":"019282b362875b23fb345d679a176a87bbd9464c087e1f31b08cfd7bb5130ce8"} Jul 28 17:15:24.501928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.501904 2579 generic.go:356] "Generic (PLEG): container finished" podID="e62a862c-3584-4f5c-a084-0a29427876d7" containerID="993c15756e092354122cd445b6d2560b7f3f98eb50026fdbf101c888a6168152" exitCode=0 Jul 28 17:15:24.502008 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.501969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" event={"ID":"e62a862c-3584-4f5c-a084-0a29427876d7","Type":"ContainerDied","Data":"993c15756e092354122cd445b6d2560b7f3f98eb50026fdbf101c888a6168152"} Jul 28 17:15:24.638653 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.638216 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:24.721629 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:15:24.721589 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62a862c_3584_4f5c_a084_0a29427876d7.slice/crio-conmon-9dfb3b6cdaa3c54fd9e8760986de99b62d676fd8b3a77c0be5df812ca8111087.scope\": RecentStats: unable to find data in memory cache]" Jul 28 17:15:24.773295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.773246 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-util\") pod \"7c8ac905-db82-43da-812c-9bbac86ee418\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " Jul 28 17:15:24.773454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.773322 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wgpr\" (UniqueName: \"kubernetes.io/projected/7c8ac905-db82-43da-812c-9bbac86ee418-kube-api-access-6wgpr\") pod \"7c8ac905-db82-43da-812c-9bbac86ee418\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " Jul 28 17:15:24.773454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.773408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-bundle\") pod \"7c8ac905-db82-43da-812c-9bbac86ee418\" (UID: \"7c8ac905-db82-43da-812c-9bbac86ee418\") " Jul 28 17:15:24.773845 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.773821 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-bundle" (OuterVolumeSpecName: "bundle") pod "7c8ac905-db82-43da-812c-9bbac86ee418" (UID: "7c8ac905-db82-43da-812c-9bbac86ee418"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:24.775573 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.775547 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c8ac905-db82-43da-812c-9bbac86ee418-kube-api-access-6wgpr" (OuterVolumeSpecName: "kube-api-access-6wgpr") pod "7c8ac905-db82-43da-812c-9bbac86ee418" (UID: "7c8ac905-db82-43da-812c-9bbac86ee418"). InnerVolumeSpecName "kube-api-access-6wgpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:15:24.778355 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.778315 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-util" (OuterVolumeSpecName: "util") pod "7c8ac905-db82-43da-812c-9bbac86ee418" (UID: "7c8ac905-db82-43da-812c-9bbac86ee418"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:24.874829 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.874790 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:24.874829 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.874820 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7c8ac905-db82-43da-812c-9bbac86ee418-util\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:24.874829 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:24.874830 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wgpr\" (UniqueName: \"kubernetes.io/projected/7c8ac905-db82-43da-812c-9bbac86ee418-kube-api-access-6wgpr\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:25.509048 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.509010 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" Jul 28 17:15:25.509495 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.509046 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8" event={"ID":"7c8ac905-db82-43da-812c-9bbac86ee418","Type":"ContainerDied","Data":"787c07e6d86551137cca1b89b8ec61a81c5af802b6ee991db5e2c5dac4c2c307"} Jul 28 17:15:25.509495 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.509098 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="787c07e6d86551137cca1b89b8ec61a81c5af802b6ee991db5e2c5dac4c2c307" Jul 28 17:15:25.511129 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.511084 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerID="53226e7b86c1330aa7c0de94f2f008b23a982bfc5a5c674943a0db2be89b6a3a" exitCode=0 Jul 28 17:15:25.511256 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.511136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" event={"ID":"6a8fecb2-2a21-40ef-aa06-9b4c84535295","Type":"ContainerDied","Data":"53226e7b86c1330aa7c0de94f2f008b23a982bfc5a5c674943a0db2be89b6a3a"} Jul 28 17:15:25.513199 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.513176 2579 generic.go:356] "Generic (PLEG): container finished" podID="e62a862c-3584-4f5c-a084-0a29427876d7" containerID="9dfb3b6cdaa3c54fd9e8760986de99b62d676fd8b3a77c0be5df812ca8111087" exitCode=0 Jul 28 17:15:25.513317 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.513257 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" event={"ID":"e62a862c-3584-4f5c-a084-0a29427876d7","Type":"ContainerDied","Data":"9dfb3b6cdaa3c54fd9e8760986de99b62d676fd8b3a77c0be5df812ca8111087"} Jul 28 17:15:25.647608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.647584 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:25.783220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.783088 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-util\") pod \"fc612252-cc0f-4016-9b05-33cbf44cfc97\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " Jul 28 17:15:25.783220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.783175 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqcq2\" (UniqueName: \"kubernetes.io/projected/fc612252-cc0f-4016-9b05-33cbf44cfc97-kube-api-access-nqcq2\") pod \"fc612252-cc0f-4016-9b05-33cbf44cfc97\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " Jul 28 17:15:25.783454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.783246 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-bundle\") pod \"fc612252-cc0f-4016-9b05-33cbf44cfc97\" (UID: \"fc612252-cc0f-4016-9b05-33cbf44cfc97\") " Jul 28 17:15:25.783943 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.783913 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-bundle" (OuterVolumeSpecName: "bundle") pod "fc612252-cc0f-4016-9b05-33cbf44cfc97" (UID: "fc612252-cc0f-4016-9b05-33cbf44cfc97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:25.785469 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.785447 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc612252-cc0f-4016-9b05-33cbf44cfc97-kube-api-access-nqcq2" (OuterVolumeSpecName: "kube-api-access-nqcq2") pod "fc612252-cc0f-4016-9b05-33cbf44cfc97" (UID: "fc612252-cc0f-4016-9b05-33cbf44cfc97"). InnerVolumeSpecName "kube-api-access-nqcq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:15:25.788803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.788755 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-util" (OuterVolumeSpecName: "util") pod "fc612252-cc0f-4016-9b05-33cbf44cfc97" (UID: "fc612252-cc0f-4016-9b05-33cbf44cfc97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:25.884135 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.884073 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:25.884135 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.884104 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc612252-cc0f-4016-9b05-33cbf44cfc97-util\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:25.884135 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:25.884137 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqcq2\" (UniqueName: \"kubernetes.io/projected/fc612252-cc0f-4016-9b05-33cbf44cfc97-kube-api-access-nqcq2\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:26.524579 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.524547 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" Jul 28 17:15:26.524579 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.524562 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd" event={"ID":"fc612252-cc0f-4016-9b05-33cbf44cfc97","Type":"ContainerDied","Data":"20e02480e01816f6d722362556f12160828786f0b7fd9d2d18b1ae9eca9becc1"} Jul 28 17:15:26.525017 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.524640 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e02480e01816f6d722362556f12160828786f0b7fd9d2d18b1ae9eca9becc1" Jul 28 17:15:26.678570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.678546 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:26.681141 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.681121 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:26.790403 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.790307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-bundle\") pod \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " Jul 28 17:15:26.790403 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.790349 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bqw9\" (UniqueName: \"kubernetes.io/projected/6a8fecb2-2a21-40ef-aa06-9b4c84535295-kube-api-access-5bqw9\") pod \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " Jul 28 17:15:26.790635 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.790398 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-util\") pod \"e62a862c-3584-4f5c-a084-0a29427876d7\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " Jul 28 17:15:26.790635 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.790428 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-bundle\") pod \"e62a862c-3584-4f5c-a084-0a29427876d7\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " Jul 28 17:15:26.790635 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.790483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcrgd\" (UniqueName: \"kubernetes.io/projected/e62a862c-3584-4f5c-a084-0a29427876d7-kube-api-access-mcrgd\") pod \"e62a862c-3584-4f5c-a084-0a29427876d7\" (UID: \"e62a862c-3584-4f5c-a084-0a29427876d7\") " Jul 28 17:15:26.790635 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.790510 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-util\") pod \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\" (UID: \"6a8fecb2-2a21-40ef-aa06-9b4c84535295\") " Jul 28 17:15:26.790957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.790930 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-bundle" (OuterVolumeSpecName: "bundle") pod "6a8fecb2-2a21-40ef-aa06-9b4c84535295" (UID: "6a8fecb2-2a21-40ef-aa06-9b4c84535295"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:26.791067 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.791046 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-bundle" (OuterVolumeSpecName: "bundle") pod "e62a862c-3584-4f5c-a084-0a29427876d7" (UID: "e62a862c-3584-4f5c-a084-0a29427876d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:26.792520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.792501 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8fecb2-2a21-40ef-aa06-9b4c84535295-kube-api-access-5bqw9" (OuterVolumeSpecName: "kube-api-access-5bqw9") pod "6a8fecb2-2a21-40ef-aa06-9b4c84535295" (UID: "6a8fecb2-2a21-40ef-aa06-9b4c84535295"). InnerVolumeSpecName "kube-api-access-5bqw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:15:26.792804 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.792788 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62a862c-3584-4f5c-a084-0a29427876d7-kube-api-access-mcrgd" (OuterVolumeSpecName: "kube-api-access-mcrgd") pod "e62a862c-3584-4f5c-a084-0a29427876d7" (UID: "e62a862c-3584-4f5c-a084-0a29427876d7"). InnerVolumeSpecName "kube-api-access-mcrgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:15:26.796982 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.796622 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-util" (OuterVolumeSpecName: "util") pod "e62a862c-3584-4f5c-a084-0a29427876d7" (UID: "e62a862c-3584-4f5c-a084-0a29427876d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:26.856407 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.856355 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-util" (OuterVolumeSpecName: "util") pod "6a8fecb2-2a21-40ef-aa06-9b4c84535295" (UID: "6a8fecb2-2a21-40ef-aa06-9b4c84535295"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:15:26.891844 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.891814 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-util\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:26.891844 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.891841 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a8fecb2-2a21-40ef-aa06-9b4c84535295-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:26.891990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.891850 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bqw9\" (UniqueName: \"kubernetes.io/projected/6a8fecb2-2a21-40ef-aa06-9b4c84535295-kube-api-access-5bqw9\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:26.891990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.891861 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-util\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:26.891990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.891870 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e62a862c-3584-4f5c-a084-0a29427876d7-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:26.891990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:26.891878 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcrgd\" (UniqueName: \"kubernetes.io/projected/e62a862c-3584-4f5c-a084-0a29427876d7-kube-api-access-mcrgd\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:15:27.531146 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:27.531100 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" Jul 28 17:15:27.531585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:27.531098 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp" event={"ID":"6a8fecb2-2a21-40ef-aa06-9b4c84535295","Type":"ContainerDied","Data":"5cbed05198dea9ef646515a1f06fb14ae1805064e8194afeafbf85be30b7ec89"} Jul 28 17:15:27.531585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:27.531246 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cbed05198dea9ef646515a1f06fb14ae1805064e8194afeafbf85be30b7ec89" Jul 28 17:15:27.532981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:27.532953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" event={"ID":"e62a862c-3584-4f5c-a084-0a29427876d7","Type":"ContainerDied","Data":"7770862e994e5eefb59f3c6893f97ae8f0e2d251c66620d38fda70f1eb7d94ce"} Jul 28 17:15:27.533087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:27.532993 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7770862e994e5eefb59f3c6893f97ae8f0e2d251c66620d38fda70f1eb7d94ce" Jul 28 17:15:27.533087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:27.532963 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg" Jul 28 17:15:33.286492 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:33.286460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:33.898574 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:33.898536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:35.143321 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:35.143281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:35.272316 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:35.272279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:38.331922 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:38.331884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:38.438263 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:38.438226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:42.512294 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:42.512254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:44.317435 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:44.317397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:44.481920 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:44.481881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:46.720031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:46.719945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:48.332798 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:48.332764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:48.477640 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:48.477602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:48.765735 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:48.765702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:48.870367 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:48.870328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:51.180282 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:51.180247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:51.572893 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:51.572798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:52.888119 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:52.888070 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:52.995501 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:52.995469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:54.545101 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:54.545060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:15:54.645563 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:15:54.645523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:00.545365 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:00.545318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:00.646511 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:00.646467 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:05.372026 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:05.371965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:05.694070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:05.693979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:09.151139 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:09.151088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:09.280561 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:09.280519 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:09.506172 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:09.506076 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:09.607221 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:09.607177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:13.546812 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:13.546772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:13.640569 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:13.640533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:19.576808 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:19.576762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:19.674516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:19.674484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:20.763699 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:20.763664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:24.373015 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:24.372978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:24.471942 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:24.471908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:30.341416 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341378 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f99699b64-9khs8"] Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341785 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8ac905-db82-43da-812c-9bbac86ee418" containerName="pull" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341803 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8ac905-db82-43da-812c-9bbac86ee418" containerName="pull" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341818 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62a862c-3584-4f5c-a084-0a29427876d7" containerName="extract" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341826 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62a862c-3584-4f5c-a084-0a29427876d7" containerName="extract" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341842 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerName="pull" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341850 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerName="pull" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341862 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerName="util" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341870 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerName="util" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341879 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8ac905-db82-43da-812c-9bbac86ee418" containerName="util" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341886 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8ac905-db82-43da-812c-9bbac86ee418" containerName="util" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341896 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62a862c-3584-4f5c-a084-0a29427876d7" containerName="util" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341903 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62a862c-3584-4f5c-a084-0a29427876d7" containerName="util" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341912 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerName="extract" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341919 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerName="extract" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341930 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerName="extract" Jul 28 17:16:30.341934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341938 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerName="extract" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341946 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c8ac905-db82-43da-812c-9bbac86ee418" containerName="extract" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341955 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c8ac905-db82-43da-812c-9bbac86ee418" containerName="extract" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341964 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62a862c-3584-4f5c-a084-0a29427876d7" containerName="pull" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341972 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62a862c-3584-4f5c-a084-0a29427876d7" containerName="pull" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341980 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerName="pull" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.341988 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerName="pull" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.342000 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerName="util" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.342008 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerName="util" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.342086 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc612252-cc0f-4016-9b05-33cbf44cfc97" containerName="extract" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.342099 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c8ac905-db82-43da-812c-9bbac86ee418" containerName="extract" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.342126 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a8fecb2-2a21-40ef-aa06-9b4c84535295" containerName="extract" Jul 28 17:16:30.342692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.342136 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62a862c-3584-4f5c-a084-0a29427876d7" containerName="extract" Jul 28 17:16:30.345350 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.345327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.357392 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.357362 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f99699b64-9khs8"] Jul 28 17:16:30.380862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.380822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-service-ca\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.380862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.380863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-config\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.381154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.380891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-trusted-ca-bundle\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.381154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.380907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-oauth-serving-cert\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.381154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.380988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-serving-cert\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.381154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.381025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-oauth-config\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.381154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.381051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm7ps\" (UniqueName: \"kubernetes.io/projected/a763eba7-3c81-41d9-b75f-febb5f85d8eb-kube-api-access-lm7ps\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.402259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.402225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:30.463165 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.463101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:30.482273 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.482233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-service-ca\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.482388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.482279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-config\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.482388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.482309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-trusted-ca-bundle\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.482388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.482328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-oauth-serving-cert\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.482388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.482373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-serving-cert\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.482608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.482392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-oauth-config\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.482608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.482421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm7ps\" (UniqueName: \"kubernetes.io/projected/a763eba7-3c81-41d9-b75f-febb5f85d8eb-kube-api-access-lm7ps\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.483185 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.483155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-service-ca\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.483307 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.483192 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-oauth-serving-cert\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.483307 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.483237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-config\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.483414 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.483396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a763eba7-3c81-41d9-b75f-febb5f85d8eb-trusted-ca-bundle\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.484894 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.484866 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-serving-cert\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.484973 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.484926 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a763eba7-3c81-41d9-b75f-febb5f85d8eb-console-oauth-config\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.494100 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.494068 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm7ps\" (UniqueName: \"kubernetes.io/projected/a763eba7-3c81-41d9-b75f-febb5f85d8eb-kube-api-access-lm7ps\") pod \"console-f99699b64-9khs8\" (UID: \"a763eba7-3c81-41d9-b75f-febb5f85d8eb\") " pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.548137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.548074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:30.654291 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.654190 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:30.761141 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.761091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:30.824003 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:30.823977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f99699b64-9khs8"] Jul 28 17:16:30.826581 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:16:30.826555 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda763eba7_3c81_41d9_b75f_febb5f85d8eb.slice/crio-aa6319f6bf42e91e129c8cd22d607d1384fd8c8acbb1144a1600f5ba9d1ee4c4 WatchSource:0}: Error finding container aa6319f6bf42e91e129c8cd22d607d1384fd8c8acbb1144a1600f5ba9d1ee4c4: Status 404 returned error can't find the container with id aa6319f6bf42e91e129c8cd22d607d1384fd8c8acbb1144a1600f5ba9d1ee4c4 Jul 28 17:16:31.145969 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:31.145934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:31.268056 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:31.268020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:31.657614 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:31.657574 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:31.827529 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:31.827492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f99699b64-9khs8" event={"ID":"a763eba7-3c81-41d9-b75f-febb5f85d8eb","Type":"ContainerStarted","Data":"cd2d1e987b1bc4aa0888094e4232e96c30fa4ccb9a1b5502a2e9db1b67644a83"} Jul 28 17:16:31.827738 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:31.827545 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f99699b64-9khs8" event={"ID":"a763eba7-3c81-41d9-b75f-febb5f85d8eb","Type":"ContainerStarted","Data":"aa6319f6bf42e91e129c8cd22d607d1384fd8c8acbb1144a1600f5ba9d1ee4c4"} Jul 28 17:16:31.855724 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:31.855615 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f99699b64-9khs8" podStartSLOduration=1.855586892 podStartE2EDuration="1.855586892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:31.851598203 +0000 UTC m=+488.655286797" watchObservedRunningTime="2026-07-28 17:16:31.855586892 +0000 UTC m=+488.659275486" Jul 28 17:16:40.654888 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:40.654827 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:40.655490 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:40.654923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:40.660510 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:40.660471 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:40.873850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:40.873820 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f99699b64-9khs8" Jul 28 17:16:40.969933 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:40.969848 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77548c74b7-mk95z"] Jul 28 17:16:41.186810 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:41.186771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:41.533362 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:41.533325 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:41.637644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:41.637612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:42.326899 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:42.326843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:42.442966 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:42.442922 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:42.548821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:42.548778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:43.080661 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.080619 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:16:43.084060 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.084036 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.092520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.092498 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 17:16:43.093265 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.093221 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 17:16:43.093397 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.093371 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-m4mtg\"" Jul 28 17:16:43.093516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.093500 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 17:16:43.101153 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.101126 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:16:43.183756 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.183716 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:16:43.193887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.193848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/963ca571-246f-42fb-af69-a05ad6dde577-config-file\") pod \"limitador-limitador-54d6d988d5-khkq9\" (UID: \"963ca571-246f-42fb-af69-a05ad6dde577\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.194041 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.193913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6zkc\" (UniqueName: \"kubernetes.io/projected/963ca571-246f-42fb-af69-a05ad6dde577-kube-api-access-c6zkc\") pod \"limitador-limitador-54d6d988d5-khkq9\" (UID: \"963ca571-246f-42fb-af69-a05ad6dde577\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.295250 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.295189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6zkc\" (UniqueName: \"kubernetes.io/projected/963ca571-246f-42fb-af69-a05ad6dde577-kube-api-access-c6zkc\") pod \"limitador-limitador-54d6d988d5-khkq9\" (UID: \"963ca571-246f-42fb-af69-a05ad6dde577\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.295439 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.295285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/963ca571-246f-42fb-af69-a05ad6dde577-config-file\") pod \"limitador-limitador-54d6d988d5-khkq9\" (UID: \"963ca571-246f-42fb-af69-a05ad6dde577\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.295982 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.295964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/963ca571-246f-42fb-af69-a05ad6dde577-config-file\") pod \"limitador-limitador-54d6d988d5-khkq9\" (UID: \"963ca571-246f-42fb-af69-a05ad6dde577\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.328475 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.328439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6zkc\" (UniqueName: \"kubernetes.io/projected/963ca571-246f-42fb-af69-a05ad6dde577-kube-api-access-c6zkc\") pod \"limitador-limitador-54d6d988d5-khkq9\" (UID: \"963ca571-246f-42fb-af69-a05ad6dde577\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.394237 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.394204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:43.416544 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.416514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:43.567613 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.567585 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:16:43.570184 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:16:43.570161 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963ca571_246f_42fb_af69_a05ad6dde577.slice/crio-9d0b81ea2f15aef705ba93cebe3f55184d93e6e78aed4db231d19971bead6be3 WatchSource:0}: Error finding container 9d0b81ea2f15aef705ba93cebe3f55184d93e6e78aed4db231d19971bead6be3: Status 404 returned error can't find the container with id 9d0b81ea2f15aef705ba93cebe3f55184d93e6e78aed4db231d19971bead6be3 Jul 28 17:16:43.823994 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.823894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:16:43.892480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:43.892444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" event={"ID":"963ca571-246f-42fb-af69-a05ad6dde577","Type":"ContainerStarted","Data":"9d0b81ea2f15aef705ba93cebe3f55184d93e6e78aed4db231d19971bead6be3"} Jul 28 17:16:46.911546 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:46.911510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" event={"ID":"963ca571-246f-42fb-af69-a05ad6dde577","Type":"ContainerStarted","Data":"ef43142a3f49b8f73ddf6e89a8e772f7c30a7d0f8f281a8f18954542aa357c3a"} Jul 28 17:16:46.912017 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:46.911645 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:16:46.940026 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:46.939973 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" podStartSLOduration=1.031167695 podStartE2EDuration="3.939955668s" totalImagesPullingTime="2.908787973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:46.935133513 +0000 UTC m=+503.738822098" watchObservedRunningTime="2026-07-28 17:16:46.939955668 +0000 UTC m=+503.743644299" Jul 28 17:16:57.917379 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:16:57.917337 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-khkq9" Jul 28 17:17:00.209055 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:00.209008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:00.338584 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:00.338548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:00.870017 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:00.869985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:05.989572 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:05.989526 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77548c74b7-mk95z" podUID="718f5f53-35db-42d2-b445-4eccfd22e9d6" containerName="console" containerID="cri-o://8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd" gracePeriod=15 Jul 28 17:17:06.238595 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.238566 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:17:06.406445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406407 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-config\") pod \"718f5f53-35db-42d2-b445-4eccfd22e9d6\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " Jul 28 17:17:06.406649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406457 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-service-ca\") pod \"718f5f53-35db-42d2-b445-4eccfd22e9d6\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " Jul 28 17:17:06.406649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406481 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqs6n\" (UniqueName: \"kubernetes.io/projected/718f5f53-35db-42d2-b445-4eccfd22e9d6-kube-api-access-rqs6n\") pod \"718f5f53-35db-42d2-b445-4eccfd22e9d6\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " Jul 28 17:17:06.406649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406515 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-trusted-ca-bundle\") pod \"718f5f53-35db-42d2-b445-4eccfd22e9d6\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " Jul 28 17:17:06.406649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406547 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-oauth-config\") pod \"718f5f53-35db-42d2-b445-4eccfd22e9d6\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " Jul 28 17:17:06.406649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406585 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-oauth-serving-cert\") pod \"718f5f53-35db-42d2-b445-4eccfd22e9d6\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " Jul 28 17:17:06.406649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406648 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-serving-cert\") pod \"718f5f53-35db-42d2-b445-4eccfd22e9d6\" (UID: \"718f5f53-35db-42d2-b445-4eccfd22e9d6\") " Jul 28 17:17:06.406964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.406884 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-config" (OuterVolumeSpecName: "console-config") pod "718f5f53-35db-42d2-b445-4eccfd22e9d6" (UID: "718f5f53-35db-42d2-b445-4eccfd22e9d6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:17:06.407131 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.407065 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-service-ca" (OuterVolumeSpecName: "service-ca") pod "718f5f53-35db-42d2-b445-4eccfd22e9d6" (UID: "718f5f53-35db-42d2-b445-4eccfd22e9d6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:17:06.407131 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.407087 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "718f5f53-35db-42d2-b445-4eccfd22e9d6" (UID: "718f5f53-35db-42d2-b445-4eccfd22e9d6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:17:06.407281 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.407193 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "718f5f53-35db-42d2-b445-4eccfd22e9d6" (UID: "718f5f53-35db-42d2-b445-4eccfd22e9d6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:17:06.408718 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.408678 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "718f5f53-35db-42d2-b445-4eccfd22e9d6" (UID: "718f5f53-35db-42d2-b445-4eccfd22e9d6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:17:06.408822 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.408740 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "718f5f53-35db-42d2-b445-4eccfd22e9d6" (UID: "718f5f53-35db-42d2-b445-4eccfd22e9d6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:17:06.408822 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.408797 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718f5f53-35db-42d2-b445-4eccfd22e9d6-kube-api-access-rqs6n" (OuterVolumeSpecName: "kube-api-access-rqs6n") pod "718f5f53-35db-42d2-b445-4eccfd22e9d6" (UID: "718f5f53-35db-42d2-b445-4eccfd22e9d6"). InnerVolumeSpecName "kube-api-access-rqs6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:17:06.508247 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.508207 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:17:06.508247 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.508239 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:17:06.508247 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.508247 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqs6n\" (UniqueName: \"kubernetes.io/projected/718f5f53-35db-42d2-b445-4eccfd22e9d6-kube-api-access-rqs6n\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:17:06.508247 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.508257 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-trusted-ca-bundle\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:17:06.508525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.508266 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-oauth-config\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:17:06.508525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.508276 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/718f5f53-35db-42d2-b445-4eccfd22e9d6-oauth-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:17:06.508525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:06.508284 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/718f5f53-35db-42d2-b445-4eccfd22e9d6-console-serving-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:17:07.007520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.007481 2579 generic.go:356] "Generic (PLEG): container finished" podID="718f5f53-35db-42d2-b445-4eccfd22e9d6" containerID="8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd" exitCode=0 Jul 28 17:17:07.007520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.007520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77548c74b7-mk95z" event={"ID":"718f5f53-35db-42d2-b445-4eccfd22e9d6","Type":"ContainerDied","Data":"8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd"} Jul 28 17:17:07.007520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.007524 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77548c74b7-mk95z" Jul 28 17:17:07.008055 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.007560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77548c74b7-mk95z" event={"ID":"718f5f53-35db-42d2-b445-4eccfd22e9d6","Type":"ContainerDied","Data":"b4c7f26052c0b0befb328582c87807365db281e89f3bb11361d9ad4d65da8720"} Jul 28 17:17:07.008055 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.007581 2579 scope.go:117] "RemoveContainer" containerID="8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd" Jul 28 17:17:07.016530 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.016510 2579 scope.go:117] "RemoveContainer" containerID="8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd" Jul 28 17:17:07.016819 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:17:07.016792 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd\": container with ID starting with 8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd not found: ID does not exist" containerID="8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd" Jul 28 17:17:07.016871 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.016828 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd"} err="failed to get container status \"8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd\": rpc error: code = NotFound desc = could not find container \"8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd\": container with ID starting with 8d67b30ebed61ef709c58db8bd00e5cc538388c4b949a6ae40df1d4572b0a2dd not found: ID does not exist" Jul 28 17:17:07.044567 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.044528 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77548c74b7-mk95z"] Jul 28 17:17:07.052441 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.052411 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77548c74b7-mk95z"] Jul 28 17:17:07.684799 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:07.684764 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="718f5f53-35db-42d2-b445-4eccfd22e9d6" path="/var/lib/kubelet/pods/718f5f53-35db-42d2-b445-4eccfd22e9d6/volumes" Jul 28 17:17:14.275221 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:14.275179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:14.383324 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:14.383288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:22.659045 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:22.658956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:22.817931 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:22.817891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:22.923869 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:22.923783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:26.407859 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:26.407825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:26.534841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:26.534800 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:56.018291 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:56.018252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:56.227152 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:56.227117 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:56.417269 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:56.417224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:56.634835 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:56.634798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:56.940255 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:56.940218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:57.895605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:57.895558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:58.040223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:58.040187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:17:58.401547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:17:58.401511 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:18:07.331693 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:07.331651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:18:07.534845 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:07.534807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:18:23.717454 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.717416 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5"] Jul 28 17:18:23.717874 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.717712 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="718f5f53-35db-42d2-b445-4eccfd22e9d6" containerName="console" Jul 28 17:18:23.717874 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.717722 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="718f5f53-35db-42d2-b445-4eccfd22e9d6" containerName="console" Jul 28 17:18:23.717874 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.717779 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="718f5f53-35db-42d2-b445-4eccfd22e9d6" containerName="console" Jul 28 17:18:23.720654 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.720632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.724509 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.724485 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-jddxn\"" Jul 28 17:18:23.724650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.724560 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 17:18:23.725760 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.725745 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 28 17:18:23.731179 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.731152 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 17:18:23.748741 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.748588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:18:23.777816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.777773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5"] Jul 28 17:18:23.831901 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.831861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.832131 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.831920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.832131 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.831968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.832131 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.831986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.832289 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.832213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.832289 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.832236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfwh\" (UniqueName: \"kubernetes.io/projected/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-kube-api-access-kcfwh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933055 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfwh\" (UniqueName: \"kubernetes.io/projected/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-kube-api-access-kcfwh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933262 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933670 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933716 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933652 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.933760 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.933719 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.935306 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.935288 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.935662 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.935643 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.947689 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.947664 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfwh\" (UniqueName: \"kubernetes.io/projected/1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab-kube-api-access-kcfwh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5\" (UID: \"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:23.982950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.982777 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:18:23.992253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:23.983432 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:18:24.031022 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:24.030970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:24.177576 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:18:24.177540 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d6d6f46_4390_4278_8b38_5e6b2c7dd7ab.slice/crio-b3472d9efd42464d3eee997534283ef86e4fe882aa3f414417bd2d493a6fb4da WatchSource:0}: Error finding container b3472d9efd42464d3eee997534283ef86e4fe882aa3f414417bd2d493a6fb4da: Status 404 returned error can't find the container with id b3472d9efd42464d3eee997534283ef86e4fe882aa3f414417bd2d493a6fb4da Jul 28 17:18:24.180262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:24.180237 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5"] Jul 28 17:18:24.372320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:24.372280 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" event={"ID":"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab","Type":"ContainerStarted","Data":"b3472d9efd42464d3eee997534283ef86e4fe882aa3f414417bd2d493a6fb4da"} Jul 28 17:18:24.773986 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:24.773892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:18:25.378477 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:25.378436 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" event={"ID":"1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab","Type":"ContainerStarted","Data":"4512cd3f5572914bc89daa291d23bc7e6a946fa4b149a3960be1053549f0b9cf"} Jul 28 17:18:25.559091 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:25.558975 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" podStartSLOduration=1.591902618 podStartE2EDuration="2.558958615s" totalImagesPullingTime="967.055997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:25.553628958 +0000 UTC m=+602.357317561" watchObservedRunningTime="2026-07-28 17:18:25.558958615 +0000 UTC m=+602.362647265" Jul 28 17:18:33.506811 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:33.506765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:18:34.031841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:34.031794 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:34.032067 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:34.031863 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:34.044473 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:34.044438 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:34.436633 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:34.436597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5" Jul 28 17:18:34.473400 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:34.473359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:18:36.669964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:36.669916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:18:40.901499 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:40.901462 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:18:43.341873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:43.341830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:18:45.016773 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:45.016732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:18:45.602840 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:45.602746 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:18:54.735273 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:54.735227 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:18:55.958594 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:18:55.958557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:04.007726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:04.007682 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:19:08.431177 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:08.431137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:08.493415 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:08.493383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:46.960826 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:46.960776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:47.065473 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:47.065437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:48.237878 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:48.237839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:48.340875 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:48.340837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:58.677234 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:58.677189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:59.908982 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:59.908939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:19:59.970839 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:19:59.970806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:20:20.093289 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:20:20.093199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:20:20.193663 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:20:20.193625 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:20:20.310892 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:20:20.310854 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:20:32.150158 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:20:32.150121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:20:41.698182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:20:41.698145 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:21:10.300790 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:21:10.300742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:21:17.210734 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:21:17.210687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:21:57.850537 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:21:57.850440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:21:59.066259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:21:59.066223 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:21.403008 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:21.402965 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:22:32.785588 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:32.785544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:32.884197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:32.884157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:33.027028 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:33.026986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:39.393295 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:39.393254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:39.496332 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:39.496290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:39.597607 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:39.597567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:39.764177 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:39.764075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:45.791575 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:45.791532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:22:45.860226 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:22:45.860186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:23:03.135623 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:03.135586 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:23:06.554475 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:06.554431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:23:12.299164 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:12.299121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:23:21.111668 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:21.111631 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:23:24.015840 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:24.015735 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:23:24.021694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:24.019662 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:23:33.008441 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:33.008405 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:23:44.203659 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:44.203619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:23:44.501596 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:44.501500 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:23:45.654799 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:45.654763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:23:54.341122 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:23:54.341063 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:24:05.006034 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:24:05.005998 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:24:14.602264 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:24:14.602224 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:24:36.822144 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:24:36.822089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:24:47.436897 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:24:47.436808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:25:12.725348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:25:12.725305 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:25:27.200554 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:25:27.200514 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:25:38.409039 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:25:38.408997 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:25:48.597965 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:25:48.597925 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:25:58.405196 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:25:58.405151 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:26:08.100129 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:26:08.100072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:26:17.723486 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:26:17.723389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:28:05.716992 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:05.716950 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:28:17.733148 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:17.733088 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:28:24.046940 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:24.046814 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:28:24.052523 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:24.052499 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:28:27.179623 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:27.179589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:28:27.265572 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:27.265529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:28:38.248903 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:38.248859 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:28:49.511343 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:49.511304 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:28:56.322386 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:56.322343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:28:59.645027 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:28:59.644992 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:29:06.169173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:29:06.169135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:29:08.039471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:29:08.039425 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:29:19.432863 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:29:19.432780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:29:19.441095 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:29:19.441059 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:29:30.212296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:29:30.212254 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:29:39.318671 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:29:39.318631 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:29:52.848029 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:29:52.847989 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:30:00.179557 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.179513 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l"] Jul 28 17:30:00.182893 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.182869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" Jul 28 17:30:00.186208 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.186182 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 17:30:00.192840 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.192815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" Jul 28 17:30:00.257605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.257568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l"] Jul 28 17:30:00.534689 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.534665 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l"] Jul 28 17:30:00.536736 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:30:00.536709 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d046c02_d6a3_41a7_8647_4083d1dc5f82.slice/crio-4f926ba0770701fb3425c3ea1aa059b754818b7844daa251c9d72d2a1881d19f WatchSource:0}: Error finding container 4f926ba0770701fb3425c3ea1aa059b754818b7844daa251c9d72d2a1881d19f: Status 404 returned error can't find the container with id 4f926ba0770701fb3425c3ea1aa059b754818b7844daa251c9d72d2a1881d19f Jul 28 17:30:00.538559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.538540 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:30:00.638913 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:00.638875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" event={"ID":"0d046c02-d6a3-41a7-8647-4083d1dc5f82","Type":"ContainerStarted","Data":"4f926ba0770701fb3425c3ea1aa059b754818b7844daa251c9d72d2a1881d19f"} Jul 28 17:30:03.656510 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:03.656462 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" event={"ID":"0d046c02-d6a3-41a7-8647-4083d1dc5f82","Type":"ContainerStarted","Data":"515f5c52bf23b12d3e46960eb17436b326fb9d067f2c11e179865f4b6fe9d785"} Jul 28 17:30:03.678501 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:03.678445 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" podStartSLOduration=1.133243068 podStartE2EDuration="3.678428661s" totalImagesPullingTime="2.545185593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:03.676789113 +0000 UTC m=+1300.480477704" watchObservedRunningTime="2026-07-28 17:30:03.678428661 +0000 UTC m=+1300.482117250" Jul 28 17:30:04.662884 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:04.662847 2579 generic.go:356] "Generic (PLEG): container finished" podID="0d046c02-d6a3-41a7-8647-4083d1dc5f82" containerID="515f5c52bf23b12d3e46960eb17436b326fb9d067f2c11e179865f4b6fe9d785" exitCode=0 Jul 28 17:30:04.663317 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:04.662938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" event={"ID":"0d046c02-d6a3-41a7-8647-4083d1dc5f82","Type":"ContainerDied","Data":"515f5c52bf23b12d3e46960eb17436b326fb9d067f2c11e179865f4b6fe9d785"} Jul 28 17:30:05.795796 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:05.795762 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" Jul 28 17:30:06.673052 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:06.673017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" event={"ID":"0d046c02-d6a3-41a7-8647-4083d1dc5f82","Type":"ContainerDied","Data":"4f926ba0770701fb3425c3ea1aa059b754818b7844daa251c9d72d2a1881d19f"} Jul 28 17:30:06.673233 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:06.673072 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f926ba0770701fb3425c3ea1aa059b754818b7844daa251c9d72d2a1881d19f" Jul 28 17:30:06.673233 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:06.673025 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754330-zjx8l" Jul 28 17:30:11.210516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:11.210480 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:30:21.801646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:21.801609 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:30:27.754737 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:27.754685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:30:38.003019 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:38.002977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:30:46.629659 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:46.629572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:30:49.600196 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:30:49.600153 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:31:07.130804 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:31:07.130762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:31:08.263072 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:31:08.263039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:31:10.608347 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:31:10.608306 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:31:24.206984 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:31:24.206943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:31:33.516656 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:31:33.516619 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:31:46.611613 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:31:46.611571 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:32:00.928247 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:32:00.928204 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:32:10.303955 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:32:10.303913 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:32:24.147270 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:32:24.147183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:32:34.026020 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:32:34.025982 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:32:47.211795 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:32:47.211753 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:32:57.099509 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:32:57.099471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:33:07.039645 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:33:07.039596 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:33:20.003390 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:33:20.003350 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:33:24.075611 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:33:24.075483 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:33:24.082313 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:33:24.082283 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:33:29.315087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:33:29.315050 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:33:35.994816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:33:35.994769 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:33:52.301592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:33:52.301510 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:34:04.699049 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:04.699001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:34:18.903201 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:18.903162 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:34:21.317332 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:21.317289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:32.213222 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:32.213179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:34:42.035594 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:42.035555 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:34:42.435844 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:42.435807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:45.753568 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:45.753525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:45.852605 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:45.852570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:46.270618 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:46.270587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:52.101520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:52.101477 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:34:52.775012 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:52.774974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:52.871801 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:52.871762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:53.051248 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:53.051160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:53.400925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:53.400889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:53.767856 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:53.767772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:55.799081 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:55.799046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:55.896359 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:55.896310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:55.992596 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:55.992555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:56.391885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:56.391851 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:56.621384 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:56.621342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:34:56.720485 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:34:56.720396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:02.897981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:02.897943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:35:21.471475 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.471389 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c"] Jul 28 17:35:21.471880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.471777 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d046c02-d6a3-41a7-8647-4083d1dc5f82" containerName="cleanup" Jul 28 17:35:21.471880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.471791 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d046c02-d6a3-41a7-8647-4083d1dc5f82" containerName="cleanup" Jul 28 17:35:21.471880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.471854 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d046c02-d6a3-41a7-8647-4083d1dc5f82" containerName="cleanup" Jul 28 17:35:21.474804 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.474784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" Jul 28 17:35:21.477345 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.477320 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 17:35:21.477482 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.477344 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 17:35:21.485182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.485152 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c"] Jul 28 17:35:21.502835 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.502791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ae2f3-8a46-40b9-90b8-9502eb6de01f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-949b0a76-tb56c\" (UID: \"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" Jul 28 17:35:21.604083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.604046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ae2f3-8a46-40b9-90b8-9502eb6de01f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-949b0a76-tb56c\" (UID: \"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" Jul 28 17:35:21.604712 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.604687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ae2f3-8a46-40b9-90b8-9502eb6de01f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-949b0a76-tb56c\" (UID: \"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" Jul 28 17:35:21.786190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.786055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" Jul 28 17:35:21.923959 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:21.923931 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c"] Jul 28 17:35:21.926132 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:35:21.926088 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8ae2f3_8a46_40b9_90b8_9502eb6de01f.slice/crio-a83b438674aab93153b7621b3132d9e72499b7045238d9a2b13b7599eeb5b758 WatchSource:0}: Error finding container a83b438674aab93153b7621b3132d9e72499b7045238d9a2b13b7599eeb5b758: Status 404 returned error can't find the container with id a83b438674aab93153b7621b3132d9e72499b7045238d9a2b13b7599eeb5b758 Jul 28 17:35:22.159944 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:22.159901 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" event={"ID":"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f","Type":"ContainerStarted","Data":"a83b438674aab93153b7621b3132d9e72499b7045238d9a2b13b7599eeb5b758"} Jul 28 17:35:23.166594 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:23.166561 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d8ae2f3-8a46-40b9-90b8-9502eb6de01f" containerID="b8b69a4dbd86cc45e454b95270bc421d726678466a03ec6e403103d085850077" exitCode=0 Jul 28 17:35:23.167080 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:23.166650 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" event={"ID":"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f","Type":"ContainerDied","Data":"b8b69a4dbd86cc45e454b95270bc421d726678466a03ec6e403103d085850077"} Jul 28 17:35:24.294386 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:24.294360 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" Jul 28 17:35:24.322350 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:24.322309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ae2f3-8a46-40b9-90b8-9502eb6de01f-maas-api-service-ca\") pod \"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f\" (UID: \"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f\") " Jul 28 17:35:24.322680 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:24.322655 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8ae2f3-8a46-40b9-90b8-9502eb6de01f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d8ae2f3-8a46-40b9-90b8-9502eb6de01f" (UID: "3d8ae2f3-8a46-40b9-90b8-9502eb6de01f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:35:24.422941 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:24.422849 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8ae2f3-8a46-40b9-90b8-9502eb6de01f-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:35:25.177526 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:25.177492 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" Jul 28 17:35:25.177682 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:25.177524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c" event={"ID":"3d8ae2f3-8a46-40b9-90b8-9502eb6de01f","Type":"ContainerDied","Data":"a83b438674aab93153b7621b3132d9e72499b7045238d9a2b13b7599eeb5b758"} Jul 28 17:35:25.177682 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:25.177596 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a83b438674aab93153b7621b3132d9e72499b7045238d9a2b13b7599eeb5b758" Jul 28 17:35:32.142865 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:32.142822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:32.247273 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:32.247242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:32.324409 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:32.324374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:35.202640 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:35.202600 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:35:36.565087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:36.565048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:36.565742 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:36.565717 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c"] Jul 28 17:35:36.572151 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:36.572097 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-949b0a76-tb56c"] Jul 28 17:35:36.800290 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:36.800258 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:37.245325 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:37.245291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:37.689424 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:37.689387 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8ae2f3-8a46-40b9-90b8-9502eb6de01f" path="/var/lib/kubelet/pods/3d8ae2f3-8a46-40b9-90b8-9502eb6de01f/volumes" Jul 28 17:35:37.805444 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:37.805410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:37.911327 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:37.911289 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:38.046755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.046659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:38.411425 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.411389 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p"] Jul 28 17:35:38.411691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.411679 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8ae2f3-8a46-40b9-90b8-9502eb6de01f" containerName="revoke-keys" Jul 28 17:35:38.411738 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.411692 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8ae2f3-8a46-40b9-90b8-9502eb6de01f" containerName="revoke-keys" Jul 28 17:35:38.411774 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.411756 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8ae2f3-8a46-40b9-90b8-9502eb6de01f" containerName="revoke-keys" Jul 28 17:35:38.415006 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.414990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.418401 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.418376 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7053c062-gw-openshift-default-dockercfg-x7p8p\"" Jul 28 17:35:38.429833 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.429804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-certs\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.429964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.429840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblng\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-kube-api-access-zblng\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.429964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.429868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-podinfo\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.429964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.429927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-envoy\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.429964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.429959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-credential-socket\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.430102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.429981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-data\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.430102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.430007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-socket\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.430102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.430041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-token\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.430102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.430070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istiod-ca-cert\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.448507 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.448472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:38.485402 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.485353 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p"] Jul 28 17:35:38.531320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-certs\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblng\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-kube-api-access-zblng\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-podinfo\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-envoy\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531418 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-credential-socket\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-data\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-socket\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-token\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istiod-ca-cert\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.531960 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.531793 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-certs\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.532035 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.532011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-credential-socket\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.532102 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.532060 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-socket\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.532275 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.532253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-data\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.532349 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.532326 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istiod-ca-cert\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.533988 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.533964 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-podinfo\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.534081 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.534017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-envoy\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.542261 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.542237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-token\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.542533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.542511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblng\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-kube-api-access-zblng\") pod \"e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.725123 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.725020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:38.729572 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.729331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:38.874357 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.874319 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p"] Jul 28 17:35:38.875594 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:35:38.875568 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2680437_8e4b_4b0b_a11e_8608b85c4ee1.slice/crio-ca270b0b18e660a0b102fb4304e595be128fa8264207966626289b930f6d3902 WatchSource:0}: Error finding container ca270b0b18e660a0b102fb4304e595be128fa8264207966626289b930f6d3902: Status 404 returned error can't find the container with id ca270b0b18e660a0b102fb4304e595be128fa8264207966626289b930f6d3902 Jul 28 17:35:38.877621 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:38.877600 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:35:39.248860 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:39.248824 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" event={"ID":"d2680437-8e4b-4b0b-a11e-8608b85c4ee1","Type":"ContainerStarted","Data":"ca270b0b18e660a0b102fb4304e595be128fa8264207966626289b930f6d3902"} Jul 28 17:35:41.894457 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:41.894420 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:35:41.894713 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:41.894496 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:35:41.894713 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:41.894523 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:35:42.267883 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:42.267779 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" event={"ID":"d2680437-8e4b-4b0b-a11e-8608b85c4ee1","Type":"ContainerStarted","Data":"52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668"} Jul 28 17:35:42.291676 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:42.291616 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" podStartSLOduration=1.275299896 podStartE2EDuration="4.291592161s" totalImagesPullingTime="3.016292265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:42.289999702 +0000 UTC m=+1639.093688306" watchObservedRunningTime="2026-07-28 17:35:42.291592161 +0000 UTC m=+1639.095280753" Jul 28 17:35:42.726018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:42.725982 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:42.730991 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:42.730960 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:43.272586 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:43.272544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:43.273666 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:43.273649 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:44.305638 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:44.305599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:44.434408 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:44.434372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:44.689893 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:44.689853 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:45.238164 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:45.238132 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:45.328354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:45.328320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:35:47.600517 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:47.600473 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:35:56.336534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:56.336494 2579 generic.go:356] "Generic (PLEG): container finished" podID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerID="52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668" exitCode=0 Jul 28 17:35:56.337043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:56.336568 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" event={"ID":"d2680437-8e4b-4b0b-a11e-8608b85c4ee1","Type":"ContainerDied","Data":"52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668"} Jul 28 17:35:56.337043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:56.336978 2579 scope.go:117] "RemoveContainer" containerID="52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668" Jul 28 17:35:56.337992 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:56.337953 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:35:56.338066 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:56.338051 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:35:56.338140 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:56.338102 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:35:57.347432 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:57.347394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" event={"ID":"d2680437-8e4b-4b0b-a11e-8608b85c4ee1","Type":"ContainerStarted","Data":"f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490"} Jul 28 17:35:57.725902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:57.725795 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:57.727184 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:57.727144 2579 patch_prober.go:28] interesting pod/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 28 17:35:57.727509 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:57.727214 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 28 17:35:58.273301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:58.273263 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:35:58.725889 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:58.725843 2579 patch_prober.go:28] interesting pod/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 28 17:35:58.726458 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:58.725933 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 28 17:35:59.725904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:59.725864 2579 patch_prober.go:28] interesting pod/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 28 17:35:59.726549 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:35:59.726516 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 28 17:36:00.726384 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:00.726340 2579 patch_prober.go:28] interesting pod/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 28 17:36:00.727033 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:00.726996 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 28 17:36:01.729713 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:01.729678 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:36:01.730680 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:01.730661 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:36:09.470417 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.470379 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv"] Jul 28 17:36:09.478426 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.478393 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" Jul 28 17:36:09.481462 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.481435 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 17:36:09.482258 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.482235 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 17:36:09.484230 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.484197 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv"] Jul 28 17:36:09.608954 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.608903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0c886-3ce0-4ac7-b8d6-0d93d921747e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv\" (UID: \"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" Jul 28 17:36:09.710209 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.710162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0c886-3ce0-4ac7-b8d6-0d93d921747e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv\" (UID: \"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" Jul 28 17:36:09.710818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.710795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0c886-3ce0-4ac7-b8d6-0d93d921747e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv\" (UID: \"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" Jul 28 17:36:09.792862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.792772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" Jul 28 17:36:09.935788 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:09.935747 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv"] Jul 28 17:36:09.937971 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:36:09.937923 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a0c886_3ce0_4ac7_b8d6_0d93d921747e.slice/crio-aae000059245ddbdd5eb9f7bdb963ef1f8bc96c061019902fa9452a6bbce8e17 WatchSource:0}: Error finding container aae000059245ddbdd5eb9f7bdb963ef1f8bc96c061019902fa9452a6bbce8e17: Status 404 returned error can't find the container with id aae000059245ddbdd5eb9f7bdb963ef1f8bc96c061019902fa9452a6bbce8e17 Jul 28 17:36:10.412617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:10.412504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" event={"ID":"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e","Type":"ContainerStarted","Data":"c6ca7a017447abdf3ab277a666cc0436c3c00442c39c82df2c5755b6f5669f90"} Jul 28 17:36:10.412617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:10.412580 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" event={"ID":"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e","Type":"ContainerStarted","Data":"aae000059245ddbdd5eb9f7bdb963ef1f8bc96c061019902fa9452a6bbce8e17"} Jul 28 17:36:10.434170 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:10.434096 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" podStartSLOduration=1.434073419 podStartE2EDuration="1.434073419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:10.432825297 +0000 UTC m=+1667.236513889" watchObservedRunningTime="2026-07-28 17:36:10.434073419 +0000 UTC m=+1667.237762008" Jul 28 17:36:11.418851 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:11.418813 2579 generic.go:356] "Generic (PLEG): container finished" podID="b4a0c886-3ce0-4ac7-b8d6-0d93d921747e" containerID="c6ca7a017447abdf3ab277a666cc0436c3c00442c39c82df2c5755b6f5669f90" exitCode=0 Jul 28 17:36:11.419274 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:11.418894 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" event={"ID":"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e","Type":"ContainerDied","Data":"c6ca7a017447abdf3ab277a666cc0436c3c00442c39c82df2c5755b6f5669f90"} Jul 28 17:36:12.554438 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:12.554411 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" Jul 28 17:36:12.634667 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:12.634624 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0c886-3ce0-4ac7-b8d6-0d93d921747e-maas-api-service-ca\") pod \"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e\" (UID: \"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e\") " Jul 28 17:36:12.635021 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:12.634997 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a0c886-3ce0-4ac7-b8d6-0d93d921747e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4a0c886-3ce0-4ac7-b8d6-0d93d921747e" (UID: "b4a0c886-3ce0-4ac7-b8d6-0d93d921747e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:36:12.735883 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:12.735780 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4a0c886-3ce0-4ac7-b8d6-0d93d921747e-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:13.429943 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:13.429909 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" Jul 28 17:36:13.429943 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:13.429932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv" event={"ID":"b4a0c886-3ce0-4ac7-b8d6-0d93d921747e","Type":"ContainerDied","Data":"aae000059245ddbdd5eb9f7bdb963ef1f8bc96c061019902fa9452a6bbce8e17"} Jul 28 17:36:13.430185 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:13.429982 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aae000059245ddbdd5eb9f7bdb963ef1f8bc96c061019902fa9452a6bbce8e17" Jul 28 17:36:19.996306 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:19.996273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:20.143244 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:20.143205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:20.244540 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:20.244500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:22.806857 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:22.806820 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:36:24.556766 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:24.556728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:24.562021 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:24.561983 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv"] Jul 28 17:36:24.564453 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:24.564419 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7053c062-8l7vv"] Jul 28 17:36:24.707646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:24.707606 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p"] Jul 28 17:36:24.707887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:24.707839 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" containerID="cri-o://f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490" gracePeriod=30 Jul 28 17:36:24.735905 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:24.735867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:25.003201 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.003162 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:25.248987 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.248950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:25.567935 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.567899 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p"] Jul 28 17:36:25.568334 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.568227 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a0c886-3ce0-4ac7-b8d6-0d93d921747e" containerName="revoke-keys" Jul 28 17:36:25.568334 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.568238 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a0c886-3ce0-4ac7-b8d6-0d93d921747e" containerName="revoke-keys" Jul 28 17:36:25.568444 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.568347 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a0c886-3ce0-4ac7-b8d6-0d93d921747e" containerName="revoke-keys" Jul 28 17:36:25.571497 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.571477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.574388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.574361 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-133ebd2a-openshift-default-dockercfg-lzdzc\"" Jul 28 17:36:25.584680 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.584652 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p"] Jul 28 17:36:25.599558 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.599526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:25.645084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-socket\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645084 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-podinfo\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istiod-ca-cert\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-token\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-credential-socket\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-envoy\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645269 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-data\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjc2f\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-kube-api-access-sjc2f\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.645348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.645337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-certs\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.686484 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.686444 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a0c886-3ce0-4ac7-b8d6-0d93d921747e" path="/var/lib/kubelet/pods/b4a0c886-3ce0-4ac7-b8d6-0d93d921747e/volumes" Jul 28 17:36:25.746266 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-token\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746477 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-credential-socket\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746477 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-envoy\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746477 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-data\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746477 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjc2f\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-kube-api-access-sjc2f\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746477 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-certs\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746768 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-socket\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746768 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-podinfo\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746768 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istiod-ca-cert\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746976 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746763 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-credential-socket\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.746976 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746907 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-data\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.747081 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.746987 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-socket\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.747190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.747160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-certs\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.747416 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.747396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istiod-ca-cert\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.748756 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.748726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-envoy\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.749133 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.749090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-podinfo\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.756015 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.755988 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-token\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.756181 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.756161 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjc2f\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-kube-api-access-sjc2f\") pod \"e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:25.882320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:25.882262 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:26.027880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.027849 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p"] Jul 28 17:36:26.030003 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:36:26.029973 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe686bc_7a5f_48dd_a8a8_1e399ae9abc8.slice/crio-830ac98a5b6cc1b6a421ef3fb2b1c7e5cb75d89a518ed7038abc38ba50a96846 WatchSource:0}: Error finding container 830ac98a5b6cc1b6a421ef3fb2b1c7e5cb75d89a518ed7038abc38ba50a96846: Status 404 returned error can't find the container with id 830ac98a5b6cc1b6a421ef3fb2b1c7e5cb75d89a518ed7038abc38ba50a96846 Jul 28 17:36:26.032412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.032379 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:36:26.032514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.032463 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:36:26.032514 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.032505 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:36:26.158447 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.158328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:26.497316 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.497229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" event={"ID":"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8","Type":"ContainerStarted","Data":"abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87"} Jul 28 17:36:26.497316 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.497286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" event={"ID":"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8","Type":"ContainerStarted","Data":"830ac98a5b6cc1b6a421ef3fb2b1c7e5cb75d89a518ed7038abc38ba50a96846"} Jul 28 17:36:26.538079 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.538024 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" podStartSLOduration=1.5380047829999999 podStartE2EDuration="1.538004783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:26.535831551 +0000 UTC m=+1683.339520140" watchObservedRunningTime="2026-07-28 17:36:26.538004783 +0000 UTC m=+1683.341693444" Jul 28 17:36:26.883178 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:26.883140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:27.888642 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:27.888609 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:27.946216 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:27.946180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:28.034245 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:28.034213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:28.106736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:28.106683 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:28.211941 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:28.211849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:28.511223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:28.511141 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:28.512044 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:28.512022 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:28.638991 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:28.638943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:28.739764 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:28.739727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:36:29.955690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.955662 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:36:29.983157 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983054 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-credential-socket\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983157 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983095 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istiod-ca-cert\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983177 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-token\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983218 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-podinfo\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983273 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-certs\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-envoy\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983327 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-socket\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983357 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zblng\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-kube-api-access-zblng\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983365 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:29.983445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-data\") pod \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\" (UID: \"d2680437-8e4b-4b0b-a11e-8608b85c4ee1\") " Jul 28 17:36:29.983887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983548 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:29.983887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983599 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:29.983887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983768 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:29.983887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983787 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:29.983887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983802 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:29.984185 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.983965 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:36:29.984403 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.984380 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:29.985854 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.985829 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:36:29.985963 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.985936 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-kube-api-access-zblng" (OuterVolumeSpecName: "kube-api-access-zblng") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "kube-api-access-zblng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:36:29.986030 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.985988 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:36:29.986030 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:29.985992 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2680437-8e4b-4b0b-a11e-8608b85c4ee1" (UID: "d2680437-8e4b-4b0b-a11e-8608b85c4ee1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:36:30.084651 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.084610 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:30.084651 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.084645 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblng\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-kube-api-access-zblng\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:30.084651 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.084654 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:30.084651 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.084664 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:30.084964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.084672 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:30.084964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.084680 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2680437-8e4b-4b0b-a11e-8608b85c4ee1-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:36:30.523714 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.523675 2579 generic.go:356] "Generic (PLEG): container finished" podID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerID="f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490" exitCode=0 Jul 28 17:36:30.523938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.523727 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" Jul 28 17:36:30.523938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.523762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" event={"ID":"d2680437-8e4b-4b0b-a11e-8608b85c4ee1","Type":"ContainerDied","Data":"f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490"} Jul 28 17:36:30.523938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.523831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p" event={"ID":"d2680437-8e4b-4b0b-a11e-8608b85c4ee1","Type":"ContainerDied","Data":"ca270b0b18e660a0b102fb4304e595be128fa8264207966626289b930f6d3902"} Jul 28 17:36:30.523938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.523865 2579 scope.go:117] "RemoveContainer" containerID="f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490" Jul 28 17:36:30.533240 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.533216 2579 scope.go:117] "RemoveContainer" containerID="52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668" Jul 28 17:36:30.541575 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.541554 2579 scope.go:117] "RemoveContainer" containerID="f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490" Jul 28 17:36:30.541871 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:36:30.541850 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490\": container with ID starting with f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490 not found: ID does not exist" containerID="f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490" Jul 28 17:36:30.541911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.541882 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490"} err="failed to get container status \"f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490\": rpc error: code = NotFound desc = could not find container \"f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490\": container with ID starting with f13a924c06cc32879848e9e735807ee14eb17648349a2d9bf0bfdb65d5efe490 not found: ID does not exist" Jul 28 17:36:30.541911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.541902 2579 scope.go:117] "RemoveContainer" containerID="52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668" Jul 28 17:36:30.542181 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:36:30.542166 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668\": container with ID starting with 52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668 not found: ID does not exist" containerID="52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668" Jul 28 17:36:30.542224 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.542185 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668"} err="failed to get container status \"52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668\": rpc error: code = NotFound desc = could not find container \"52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668\": container with ID starting with 52b39ea9a1e7b46252701249a3366d9d575a9aff44c2db4971f179a011fd9668 not found: ID does not exist" Jul 28 17:36:30.548104 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.548068 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p"] Jul 28 17:36:30.553003 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:30.552973 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7053c062-gw-openshift-default-758b856f6-f6s2p"] Jul 28 17:36:31.683808 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:31.683770 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" path="/var/lib/kubelet/pods/d2680437-8e4b-4b0b-a11e-8608b85c4ee1/volumes" Jul 28 17:36:36.218134 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:36.205499 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:36:43.809552 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:43.809514 2579 patch_prober.go:28] interesting pod/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:40964->10.133.0.39:15021: read: connection reset by peer" start-of-body= Jul 28 17:36:43.809941 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:43.809604 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": read tcp 10.133.0.2:40964->10.133.0.39:15021: read: connection reset by peer" Jul 28 17:36:44.595957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:44.595920 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerID="abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87" exitCode=0 Jul 28 17:36:44.596201 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:44.595995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" event={"ID":"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8","Type":"ContainerDied","Data":"abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87"} Jul 28 17:36:44.596496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:44.596479 2579 scope.go:117] "RemoveContainer" containerID="abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87" Jul 28 17:36:44.597491 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:44.597468 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:36:44.597550 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:44.597535 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:36:44.597590 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:44.597566 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:36:45.602383 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:45.602302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" event={"ID":"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8","Type":"ContainerStarted","Data":"d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8"} Jul 28 17:36:45.882502 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:45.882395 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:45.882740 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:45.882712 2579 patch_prober.go:28] interesting pod/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 28 17:36:45.882797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:45.882779 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 28 17:36:46.883248 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:46.883205 2579 patch_prober.go:28] interesting pod/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 28 17:36:46.883664 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:46.883265 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 28 17:36:47.883548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:47.883507 2579 patch_prober.go:28] interesting pod/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 28 17:36:47.883957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:47.883571 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 28 17:36:48.883183 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:48.883137 2579 patch_prober.go:28] interesting pod/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 28 17:36:48.883380 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:48.883200 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 28 17:36:49.886737 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:49.886698 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:49.887159 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:49.887073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:36:49.887873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:36:49.887855 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:37:04.592873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:04.592835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:04.691401 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:04.691347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:04.792345 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:04.792310 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:07.501429 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:07.501385 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:37:08.955690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:08.955654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:10.201124 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:10.201074 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p"] Jul 28 17:37:10.201596 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:10.201341 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" containerID="cri-o://d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8" gracePeriod=30 Jul 28 17:37:10.231335 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:10.231301 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:10.477556 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:10.477465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:10.716494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:10.716458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:11.025384 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:11.025351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:11.122695 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:11.122658 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:13.560256 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:13.560219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:13.644587 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:13.644553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:13.722387 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:13.722356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:13.879294 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:13.879259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:14.377951 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:14.377903 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:14.485023 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:14.484979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:15.463389 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.463320 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:37:15.596182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596139 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-token\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596342 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596206 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-socket\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596342 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596243 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-certs\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596342 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596287 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-credential-socket\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596342 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596313 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjc2f\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-kube-api-access-sjc2f\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596365 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istiod-ca-cert\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596398 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-podinfo\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596454 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-data\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596489 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-envoy\") pod \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\" (UID: \"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8\") " Jul 28 17:37:15.596570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596556 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:15.596836 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596570 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:15.596836 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596786 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.596836 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596805 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.596994 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596839 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:15.596994 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.596959 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:15.597426 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.597404 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:15.598445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.598417 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:37:15.598543 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.598525 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-kube-api-access-sjc2f" (OuterVolumeSpecName: "kube-api-access-sjc2f") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "kube-api-access-sjc2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:37:15.598961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.598940 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:37:15.598961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.598948 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" (UID: "bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:37:15.698120 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.698083 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.698253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.698123 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjc2f\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-kube-api-access-sjc2f\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.698253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.698138 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.698253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.698150 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.698253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.698162 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.698253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.698170 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.698253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.698179 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:15.753393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.753295 2579 generic.go:356] "Generic (PLEG): container finished" podID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerID="d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8" exitCode=0 Jul 28 17:37:15.753393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.753349 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" Jul 28 17:37:15.753393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.753356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" event={"ID":"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8","Type":"ContainerDied","Data":"d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8"} Jul 28 17:37:15.753684 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.753401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p" event={"ID":"bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8","Type":"ContainerDied","Data":"830ac98a5b6cc1b6a421ef3fb2b1c7e5cb75d89a518ed7038abc38ba50a96846"} Jul 28 17:37:15.753684 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.753422 2579 scope.go:117] "RemoveContainer" containerID="d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8" Jul 28 17:37:15.762404 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.762386 2579 scope.go:117] "RemoveContainer" containerID="abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87" Jul 28 17:37:15.770638 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.770619 2579 scope.go:117] "RemoveContainer" containerID="d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8" Jul 28 17:37:15.770917 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:37:15.770898 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8\": container with ID starting with d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8 not found: ID does not exist" containerID="d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8" Jul 28 17:37:15.770960 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.770930 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8"} err="failed to get container status \"d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8\": rpc error: code = NotFound desc = could not find container \"d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8\": container with ID starting with d7ee40b323c4a97919529f34c7ced38e59e9b0733658ba89ac54a17c4bef2eb8 not found: ID does not exist" Jul 28 17:37:15.770960 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.770953 2579 scope.go:117] "RemoveContainer" containerID="abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87" Jul 28 17:37:15.771225 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:37:15.771207 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87\": container with ID starting with abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87 not found: ID does not exist" containerID="abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87" Jul 28 17:37:15.771268 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.771236 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87"} err="failed to get container status \"abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87\": rpc error: code = NotFound desc = could not find container \"abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87\": container with ID starting with abeda82268bd76805cae965473225b1e642e7999e07128f2a13e9f7a13fb5d87 not found: ID does not exist" Jul 28 17:37:15.775282 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.775258 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p"] Jul 28 17:37:15.779530 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:15.779504 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-133ebd2a-openshift-default-5598794b94-btd4p"] Jul 28 17:37:17.684696 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:17.684652 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" path="/var/lib/kubelet/pods/bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8/volumes" Jul 28 17:37:18.401052 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:18.401001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:37:38.491366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491325 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc"] Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491611 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491622 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491641 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491646 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491652 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491657 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491707 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491715 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbe686bc-7a5f-48dd-a8a8-1e399ae9abc8" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491724 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" Jul 28 17:37:38.491885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.491730 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" Jul 28 17:37:38.494302 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.494285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" Jul 28 17:37:38.497380 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.497360 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 17:37:38.497613 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.497599 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 17:37:38.509974 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.509937 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc"] Jul 28 17:37:38.580122 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.580061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37518138-1fa8-43b9-a808-8b4d1b0cc22f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc\" (UID: \"37518138-1fa8-43b9-a808-8b4d1b0cc22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" Jul 28 17:37:38.680928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.680875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37518138-1fa8-43b9-a808-8b4d1b0cc22f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc\" (UID: \"37518138-1fa8-43b9-a808-8b4d1b0cc22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" Jul 28 17:37:38.681532 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.681510 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37518138-1fa8-43b9-a808-8b4d1b0cc22f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc\" (UID: \"37518138-1fa8-43b9-a808-8b4d1b0cc22f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" Jul 28 17:37:38.809662 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.809559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" Jul 28 17:37:38.953988 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:38.953906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc"] Jul 28 17:37:38.956177 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:37:38.956145 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37518138_1fa8_43b9_a808_8b4d1b0cc22f.slice/crio-72abb7fb57871ab58b9b35c94c2e73b850decf0306b3e7e0fc162ae222089904 WatchSource:0}: Error finding container 72abb7fb57871ab58b9b35c94c2e73b850decf0306b3e7e0fc162ae222089904: Status 404 returned error can't find the container with id 72abb7fb57871ab58b9b35c94c2e73b850decf0306b3e7e0fc162ae222089904 Jul 28 17:37:39.871709 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:39.871676 2579 generic.go:356] "Generic (PLEG): container finished" podID="37518138-1fa8-43b9-a808-8b4d1b0cc22f" containerID="2034733444ff94d5f0cf98f18b5cd2fecf4a20333674975cf9b58dd1910ca1f1" exitCode=0 Jul 28 17:37:39.872147 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:39.871750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" event={"ID":"37518138-1fa8-43b9-a808-8b4d1b0cc22f","Type":"ContainerDied","Data":"2034733444ff94d5f0cf98f18b5cd2fecf4a20333674975cf9b58dd1910ca1f1"} Jul 28 17:37:39.872147 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:39.871794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" event={"ID":"37518138-1fa8-43b9-a808-8b4d1b0cc22f","Type":"ContainerStarted","Data":"72abb7fb57871ab58b9b35c94c2e73b850decf0306b3e7e0fc162ae222089904"} Jul 28 17:37:40.999074 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:40.999045 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" Jul 28 17:37:41.099018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:41.098972 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37518138-1fa8-43b9-a808-8b4d1b0cc22f-maas-api-service-ca\") pod \"37518138-1fa8-43b9-a808-8b4d1b0cc22f\" (UID: \"37518138-1fa8-43b9-a808-8b4d1b0cc22f\") " Jul 28 17:37:41.099390 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:41.099367 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37518138-1fa8-43b9-a808-8b4d1b0cc22f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37518138-1fa8-43b9-a808-8b4d1b0cc22f" (UID: "37518138-1fa8-43b9-a808-8b4d1b0cc22f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:37:41.200207 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:41.200097 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37518138-1fa8-43b9-a808-8b4d1b0cc22f-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:37:41.882403 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:41.882375 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" Jul 28 17:37:41.882578 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:41.882375 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc" event={"ID":"37518138-1fa8-43b9-a808-8b4d1b0cc22f","Type":"ContainerDied","Data":"72abb7fb57871ab58b9b35c94c2e73b850decf0306b3e7e0fc162ae222089904"} Jul 28 17:37:41.882578 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:41.882497 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72abb7fb57871ab58b9b35c94c2e73b850decf0306b3e7e0fc162ae222089904" Jul 28 17:37:42.288877 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:42.288841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:42.389254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:42.389218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:42.648414 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:42.648387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:49.012325 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:49.012291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:49.084250 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:49.084217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:49.145482 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:49.145442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:52.497861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:52.497814 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:37:53.566399 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:53.566366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:53.574520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:53.574490 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc"] Jul 28 17:37:53.578138 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:53.578102 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-31d2617b-dr9qc"] Jul 28 17:37:53.684197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:53.684167 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37518138-1fa8-43b9-a808-8b4d1b0cc22f" path="/var/lib/kubelet/pods/37518138-1fa8-43b9-a808-8b4d1b0cc22f/volumes" Jul 28 17:37:53.748670 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:53.748633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:53.997187 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:53.997139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:54.548734 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:54.548699 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:54.643383 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:54.643343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:54.743525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:54.743490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:37:56.420811 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:37:56.420767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:00.495414 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:00.495380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:00.596551 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:00.596517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:02.482524 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:02.482486 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:02.583266 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:02.583228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:02.682829 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:02.682782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:02.808222 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:02.808136 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:03.292162 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:03.292130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:24.107429 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:24.107393 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:38:24.115981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:24.115953 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:38:29.314718 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:29.314685 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:36.569555 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:36.569520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:36.643930 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:36.643890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:36.728701 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:36.728660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:52.739435 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:52.739399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:58.917786 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:58.917748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:59.018666 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:59.018633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:38:59.117452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:38:59.117422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:09.787005 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:09.786974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:10.705935 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:10.705894 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:17.946573 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:17.946536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:18.047809 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:18.047770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:18.159460 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:18.159410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:18.253148 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:18.253027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:19.151589 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:19.151556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:19.250517 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:19.250485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:19.350144 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:19.350100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:21.143249 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:21.143214 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:22.203553 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:22.203509 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:39:27.457763 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:27.457729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:28.632011 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:28.631979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:34.308198 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:34.308158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:39:34.610367 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:34.610331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:34.709813 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:34.709775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:34.879227 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:34.879128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:35.285618 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.285530 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz"] Jul 28 17:39:35.285896 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.285879 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" Jul 28 17:39:35.285954 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.285899 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2680437-8e4b-4b0b-a11e-8608b85c4ee1" containerName="istio-proxy" Jul 28 17:39:35.285954 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.285925 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37518138-1fa8-43b9-a808-8b4d1b0cc22f" containerName="revoke-keys" Jul 28 17:39:35.285954 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.285933 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="37518138-1fa8-43b9-a808-8b4d1b0cc22f" containerName="revoke-keys" Jul 28 17:39:35.286066 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.286003 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="37518138-1fa8-43b9-a808-8b4d1b0cc22f" containerName="revoke-keys" Jul 28 17:39:35.286685 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.286655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:35.288994 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.288979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.292719 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.292696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d9be9a39-openshift-default-dockercfg-x5r8w\"" Jul 28 17:39:35.327593 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.327548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz"] Jul 28 17:39:35.328087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.327858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istiod-ca-cert\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.327906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-envoy\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.327932 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-data\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.327954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-podinfo\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.327997 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-certs\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.328041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-token\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.328069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkbjh\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-kube-api-access-hkbjh\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328462 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.328127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-socket\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.328462 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.328197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-credential-socket\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429311 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-credential-socket\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429311 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istiod-ca-cert\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-envoy\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-data\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-podinfo\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-certs\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-token\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkbjh\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-kube-api-access-hkbjh\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-socket\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.429953 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.429742 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-credential-socket\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.430156 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.430129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istiod-ca-cert\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.430342 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.430315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-socket\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.430435 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.430342 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-certs\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.430435 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.430374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-data\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.431864 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.431843 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-envoy\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.432063 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.432045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-podinfo\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.441570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.441540 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-token\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.441777 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.441756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkbjh\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-kube-api-access-hkbjh\") pod \"e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.499790 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.499755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:35.599320 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.599279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:35.743407 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.743380 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz"] Jul 28 17:39:35.745325 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:39:35.745295 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c8bc96_301e_485c_8e6f_eea025c6e9a6.slice/crio-1e95186e1193968422b9e68cc27a6fcc544c19d7bf5cb52490d6eed84e6e4c87 WatchSource:0}: Error finding container 1e95186e1193968422b9e68cc27a6fcc544c19d7bf5cb52490d6eed84e6e4c87: Status 404 returned error can't find the container with id 1e95186e1193968422b9e68cc27a6fcc544c19d7bf5cb52490d6eed84e6e4c87 Jul 28 17:39:35.747348 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.747314 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:39:35.747455 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.747393 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:39:35.747455 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:35.747438 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:39:36.438446 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:36.438405 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" event={"ID":"15c8bc96-301e-485c-8e6f-eea025c6e9a6","Type":"ContainerStarted","Data":"b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f"} Jul 28 17:39:36.438909 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:36.438463 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" event={"ID":"15c8bc96-301e-485c-8e6f-eea025c6e9a6","Type":"ContainerStarted","Data":"1e95186e1193968422b9e68cc27a6fcc544c19d7bf5cb52490d6eed84e6e4c87"} Jul 28 17:39:36.469265 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:36.469204 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" podStartSLOduration=1.469188594 podStartE2EDuration="1.469188594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:36.467331086 +0000 UTC m=+1873.271019723" watchObservedRunningTime="2026-07-28 17:39:36.469188594 +0000 UTC m=+1873.272877184" Jul 28 17:39:36.599911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:36.599868 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:36.604460 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:36.604429 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:37.444431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:37.444399 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:37.445412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:37.445392 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:38.507812 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:38.507771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:38.610009 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:38.609970 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:38.706748 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:38.706719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:39.069442 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:39.069409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:39.368785 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:39.368736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:39.470633 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:39.470583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:43.044803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:43.044760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:43.768287 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:43.768254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:43.863567 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:43.863536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:44.235796 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:44.235761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:47.792025 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:47.791986 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:39:49.504697 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:49.504667 2579 generic.go:356] "Generic (PLEG): container finished" podID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerID="b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f" exitCode=0 Jul 28 17:39:49.505220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:49.504748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" event={"ID":"15c8bc96-301e-485c-8e6f-eea025c6e9a6","Type":"ContainerDied","Data":"b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f"} Jul 28 17:39:49.505220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:49.505184 2579 scope.go:117] "RemoveContainer" containerID="b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f" Jul 28 17:39:49.506154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:49.506124 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:39:49.506268 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:49.506230 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:39:49.506332 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:49.506290 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:39:50.517195 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:50.517146 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" event={"ID":"15c8bc96-301e-485c-8e6f-eea025c6e9a6","Type":"ContainerStarted","Data":"ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9"} Jul 28 17:39:50.548655 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:50.548609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:50.600287 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:50.600249 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:50.601300 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:50.601269 2579 patch_prober.go:28] interesting pod/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 28 17:39:50.601429 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:50.601325 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 28 17:39:50.623412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:50.623348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:50.829275 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:50.829177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:39:51.599952 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:51.599916 2579 patch_prober.go:28] interesting pod/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 28 17:39:51.600384 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:51.599975 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 28 17:39:52.444551 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:52.444422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:52.600475 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:52.600436 2579 patch_prober.go:28] interesting pod/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 28 17:39:52.600885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:52.600491 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 28 17:39:53.600033 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:53.599997 2579 patch_prober.go:28] interesting pod/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 28 17:39:53.600253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:53.600055 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 28 17:39:54.604146 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:54.604099 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:39:54.605390 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:39:54.605368 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:40:01.120696 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:01.120653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:01.217432 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:01.217387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:01.317359 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:01.317314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:04.095633 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:04.095591 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:40:06.022879 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:06.022838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:06.433119 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:06.433079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:12.326179 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:12.326130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:12.388463 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:12.388426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:12.587687 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:12.587657 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz"] Jul 28 17:40:12.587917 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:12.587892 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" containerID="cri-o://ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9" gracePeriod=30 Jul 28 17:40:12.610515 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:12.610476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:12.854327 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:12.854237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:12.985755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:12.985718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:13.133090 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:13.133006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:13.235978 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:13.235942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:17.842654 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.842627 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:40:17.914641 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914542 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-certs\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914641 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914595 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-data\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914641 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914628 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istiod-ca-cert\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914663 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkbjh\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-kube-api-access-hkbjh\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914692 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-token\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914731 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-podinfo\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914759 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-socket\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914805 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-envoy\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914832 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-credential-socket\") pod \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\" (UID: \"15c8bc96-301e-485c-8e6f-eea025c6e9a6\") " Jul 28 17:40:17.914925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.914868 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:40:17.915338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.915102 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:40:17.915338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.915102 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:17.915338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.915208 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:40:17.915338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.915330 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:40:17.915571 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.915457 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:40:17.917234 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.917206 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-kube-api-access-hkbjh" (OuterVolumeSpecName: "kube-api-access-hkbjh") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "kube-api-access-hkbjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:40:17.917343 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.917287 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:40:17.917412 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.917398 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:40:17.917500 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:17.917477 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "15c8bc96-301e-485c-8e6f-eea025c6e9a6" (UID: "15c8bc96-301e-485c-8e6f-eea025c6e9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:40:18.015735 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015679 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.015735 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015727 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkbjh\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-kube-api-access-hkbjh\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.015735 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015740 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.015735 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015749 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.015735 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015757 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.016056 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015766 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.016056 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015775 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.016056 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.015783 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15c8bc96-301e-485c-8e6f-eea025c6e9a6-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:40:18.671244 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.671211 2579 generic.go:356] "Generic (PLEG): container finished" podID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerID="ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9" exitCode=0 Jul 28 17:40:18.671431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.671259 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" Jul 28 17:40:18.671431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.671299 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" event={"ID":"15c8bc96-301e-485c-8e6f-eea025c6e9a6","Type":"ContainerDied","Data":"ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9"} Jul 28 17:40:18.671431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.671357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz" event={"ID":"15c8bc96-301e-485c-8e6f-eea025c6e9a6","Type":"ContainerDied","Data":"1e95186e1193968422b9e68cc27a6fcc544c19d7bf5cb52490d6eed84e6e4c87"} Jul 28 17:40:18.671431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.671381 2579 scope.go:117] "RemoveContainer" containerID="ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9" Jul 28 17:40:18.688062 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.688031 2579 scope.go:117] "RemoveContainer" containerID="b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f" Jul 28 17:40:18.697891 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.697867 2579 scope.go:117] "RemoveContainer" containerID="ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9" Jul 28 17:40:18.698508 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:40:18.698465 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9\": container with ID starting with ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9 not found: ID does not exist" containerID="ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9" Jul 28 17:40:18.698628 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.698570 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9"} err="failed to get container status \"ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9\": rpc error: code = NotFound desc = could not find container \"ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9\": container with ID starting with ac99d63f7c00725896797018f918c8dbac67b16245f9012a62d7c943a57b91d9 not found: ID does not exist" Jul 28 17:40:18.698701 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.698634 2579 scope.go:117] "RemoveContainer" containerID="b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f" Jul 28 17:40:18.699020 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:40:18.698993 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f\": container with ID starting with b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f not found: ID does not exist" containerID="b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f" Jul 28 17:40:18.699092 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.699028 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f"} err="failed to get container status \"b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f\": rpc error: code = NotFound desc = could not find container \"b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f\": container with ID starting with b6ebe7005e3d745570a71cac76f2e0fe680e29d3d51e1eefe1d8b8ff62d6ef1f not found: ID does not exist" Jul 28 17:40:18.703094 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.703058 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz"] Jul 28 17:40:18.707215 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:18.707187 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d9be9a39-openshift-default-7b54d65c7b-5hdjz"] Jul 28 17:40:19.563762 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:19.563729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:19.634373 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:19.634343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:19.684354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:19.684320 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" path="/var/lib/kubelet/pods/15c8bc96-301e-485c-8e6f-eea025c6e9a6/volumes" Jul 28 17:40:19.705426 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:19.705394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:19.804977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:19.804946 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:19.940993 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:19.940953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:25.995078 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:25.995033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:26.062241 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:26.062202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:32.972691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:32.972651 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:33.285453 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.285345 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg"] Jul 28 17:40:33.285733 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.285718 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" Jul 28 17:40:33.285821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.285734 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" Jul 28 17:40:33.285821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.285748 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" Jul 28 17:40:33.285821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.285757 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" Jul 28 17:40:33.285821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.285814 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" Jul 28 17:40:33.285821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.285822 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c8bc96-301e-485c-8e6f-eea025c6e9a6" containerName="istio-proxy" Jul 28 17:40:33.287783 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.287758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.290504 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.290484 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4794d51a-openshift-default-dockercfg-2sm5p\"" Jul 28 17:40:33.299921 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.299890 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg"] Jul 28 17:40:33.316262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.316233 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:33.325658 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-certs\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.325658 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa16f-aa81-4347-b03c-43ff52c25931-istiod-ca-cert\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.325881 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-envoy\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.325881 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-token\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.325881 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-credential-socket\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.325881 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-data\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.326045 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325875 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25n9\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-kube-api-access-n25n9\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.326045 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-socket\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.326045 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.325922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-podinfo\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.426820 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.426772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-podinfo\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.426820 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.426830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-certs\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427082 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.426851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa16f-aa81-4347-b03c-43ff52c25931-istiod-ca-cert\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427177 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-envoy\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-token\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-credential-socket\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427241 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-certs\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-data\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25n9\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-kube-api-access-n25n9\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-socket\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427557 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427510 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-credential-socket\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427729 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-socket\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427852 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa16f-aa81-4347-b03c-43ff52c25931-istiod-ca-cert\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.427921 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.427851 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-data\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.429559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.429536 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-envoy\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.429797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.429778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-podinfo\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.437822 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.437795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-token\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.437926 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.437878 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25n9\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-kube-api-access-n25n9\") pod \"e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.600731 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.600698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:33.745366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.745270 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg"] Jul 28 17:40:33.748197 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:40:33.748163 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6fa16f_aa81_4347_b03c_43ff52c25931.slice/crio-308dbf9e6d5b797b835b210e6c2c91b6f6045a4c9f734c799d935fd5f5ad355e WatchSource:0}: Error finding container 308dbf9e6d5b797b835b210e6c2c91b6f6045a4c9f734c799d935fd5f5ad355e: Status 404 returned error can't find the container with id 308dbf9e6d5b797b835b210e6c2c91b6f6045a4c9f734c799d935fd5f5ad355e Jul 28 17:40:33.750580 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.750552 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:40:33.750676 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.750616 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:40:33.750676 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:33.750644 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:40:34.219380 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:34.219336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:34.755640 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:34.755602 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" event={"ID":"2e6fa16f-aa81-4347-b03c-43ff52c25931","Type":"ContainerStarted","Data":"c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96"} Jul 28 17:40:34.755834 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:34.755658 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" event={"ID":"2e6fa16f-aa81-4347-b03c-43ff52c25931","Type":"ContainerStarted","Data":"308dbf9e6d5b797b835b210e6c2c91b6f6045a4c9f734c799d935fd5f5ad355e"} Jul 28 17:40:34.780329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:34.780276 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" podStartSLOduration=1.780257593 podStartE2EDuration="1.780257593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:34.778974711 +0000 UTC m=+1931.582663327" watchObservedRunningTime="2026-07-28 17:40:34.780257593 +0000 UTC m=+1931.583946182" Jul 28 17:40:35.601672 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:35.601624 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:35.606674 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:35.606645 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:35.761059 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:35.761018 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:35.761982 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:35.761964 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:36.739304 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:36.739268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:36.831923 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:36.831888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:36.899572 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:36.899539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:37.091083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:37.091049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:37.459810 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:37.459711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:37.551146 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:37.551099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:41.297621 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:41.297581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:42.998693 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:42.998653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:40:51.115814 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.115778 2579 patch_prober.go:28] interesting pod/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:48664->10.133.0.42:15021: read: connection reset by peer" start-of-body= Jul 28 17:40:51.116262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.115867 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": read tcp 10.133.0.2:48664->10.133.0.42:15021: read: connection reset by peer" Jul 28 17:40:51.841349 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.841314 2579 generic.go:356] "Generic (PLEG): container finished" podID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerID="c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96" exitCode=0 Jul 28 17:40:51.841530 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.841388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" event={"ID":"2e6fa16f-aa81-4347-b03c-43ff52c25931","Type":"ContainerDied","Data":"c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96"} Jul 28 17:40:51.841847 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.841827 2579 scope.go:117] "RemoveContainer" containerID="c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96" Jul 28 17:40:51.842838 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.842806 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:40:51.842901 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.842876 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:40:51.842936 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:51.842908 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:40:52.850790 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:52.850753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" event={"ID":"2e6fa16f-aa81-4347-b03c-43ff52c25931","Type":"ContainerStarted","Data":"92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf"} Jul 28 17:40:53.601609 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:53.601571 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:53.601880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:53.601853 2579 patch_prober.go:28] interesting pod/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 28 17:40:53.601935 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:53.601918 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 28 17:40:53.966322 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:53.966224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:40:54.601959 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:54.601919 2579 patch_prober.go:28] interesting pod/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 28 17:40:54.602182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:54.601983 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 28 17:40:55.602162 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:55.602086 2579 patch_prober.go:28] interesting pod/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 28 17:40:55.602560 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:55.602184 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 28 17:40:56.601509 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:56.601471 2579 patch_prober.go:28] interesting pod/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 28 17:40:56.601689 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:56.601529 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 28 17:40:57.017842 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:57.017747 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:40:57.604820 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:57.604786 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:57.605078 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:57.605057 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:40:57.605875 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:40:57.605854 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:41:23.807013 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:23.806920 2579 scope.go:117] "RemoveContainer" containerID="b8b69a4dbd86cc45e454b95270bc421d726678466a03ec6e403103d085850077" Jul 28 17:41:26.703799 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:26.703758 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:41:31.188506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.188470 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s"] Jul 28 17:41:31.190640 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.190623 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" Jul 28 17:41:31.193457 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.193435 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 17:41:31.193604 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.193436 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 17:41:31.200212 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.200174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s"] Jul 28 17:41:31.258998 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.258956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e02d23-85e6-4c34-9a47-de3d85a1c2df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4794d51a-kh64s\" (UID: \"61e02d23-85e6-4c34-9a47-de3d85a1c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" Jul 28 17:41:31.360361 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.360314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e02d23-85e6-4c34-9a47-de3d85a1c2df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4794d51a-kh64s\" (UID: \"61e02d23-85e6-4c34-9a47-de3d85a1c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" Jul 28 17:41:31.361036 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.361013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e02d23-85e6-4c34-9a47-de3d85a1c2df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4794d51a-kh64s\" (UID: \"61e02d23-85e6-4c34-9a47-de3d85a1c2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" Jul 28 17:41:31.502903 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.502809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" Jul 28 17:41:31.645873 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:31.645801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s"] Jul 28 17:41:31.648326 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:41:31.648297 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61e02d23_85e6_4c34_9a47_de3d85a1c2df.slice/crio-6ae3360e5001adc2bd3aa6440a23f90896a5927d74e7be58d905d166b2efba77 WatchSource:0}: Error finding container 6ae3360e5001adc2bd3aa6440a23f90896a5927d74e7be58d905d166b2efba77: Status 404 returned error can't find the container with id 6ae3360e5001adc2bd3aa6440a23f90896a5927d74e7be58d905d166b2efba77 Jul 28 17:41:32.050083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:32.050039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" event={"ID":"61e02d23-85e6-4c34-9a47-de3d85a1c2df","Type":"ContainerStarted","Data":"b611bb809381f687479b0ae02ca292c27a2f5e14e6051740c137de3b44ca93e7"} Jul 28 17:41:32.050303 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:32.050103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" event={"ID":"61e02d23-85e6-4c34-9a47-de3d85a1c2df","Type":"ContainerStarted","Data":"6ae3360e5001adc2bd3aa6440a23f90896a5927d74e7be58d905d166b2efba77"} Jul 28 17:41:32.073226 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:32.073170 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" podStartSLOduration=1.073152111 podStartE2EDuration="1.073152111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:32.070448446 +0000 UTC m=+1988.874137036" watchObservedRunningTime="2026-07-28 17:41:32.073152111 +0000 UTC m=+1988.876840700" Jul 28 17:41:33.056536 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:33.056503 2579 generic.go:356] "Generic (PLEG): container finished" podID="61e02d23-85e6-4c34-9a47-de3d85a1c2df" containerID="b611bb809381f687479b0ae02ca292c27a2f5e14e6051740c137de3b44ca93e7" exitCode=0 Jul 28 17:41:33.056942 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:33.056578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" event={"ID":"61e02d23-85e6-4c34-9a47-de3d85a1c2df","Type":"ContainerDied","Data":"b611bb809381f687479b0ae02ca292c27a2f5e14e6051740c137de3b44ca93e7"} Jul 28 17:41:34.196736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:34.196706 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" Jul 28 17:41:34.288988 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:34.288949 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e02d23-85e6-4c34-9a47-de3d85a1c2df-maas-api-service-ca\") pod \"61e02d23-85e6-4c34-9a47-de3d85a1c2df\" (UID: \"61e02d23-85e6-4c34-9a47-de3d85a1c2df\") " Jul 28 17:41:34.289333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:34.289310 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61e02d23-85e6-4c34-9a47-de3d85a1c2df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61e02d23-85e6-4c34-9a47-de3d85a1c2df" (UID: "61e02d23-85e6-4c34-9a47-de3d85a1c2df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:41:34.390493 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:34.390450 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61e02d23-85e6-4c34-9a47-de3d85a1c2df-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:41:35.069418 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:35.069381 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" Jul 28 17:41:35.069418 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:35.069411 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s" event={"ID":"61e02d23-85e6-4c34-9a47-de3d85a1c2df","Type":"ContainerDied","Data":"6ae3360e5001adc2bd3aa6440a23f90896a5927d74e7be58d905d166b2efba77"} Jul 28 17:41:35.069642 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:35.069459 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ae3360e5001adc2bd3aa6440a23f90896a5927d74e7be58d905d166b2efba77" Jul 28 17:41:37.143750 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:37.143715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:41:45.297449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:45.297407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:41:46.803471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:46.803433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:41:46.884981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:46.884945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:41:46.968748 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:46.968712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:41:51.304715 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:51.304664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:41:51.316381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:51.316350 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s"] Jul 28 17:41:51.320847 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:51.320816 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4794d51a-kh64s"] Jul 28 17:41:51.684410 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:51.684369 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61e02d23-85e6-4c34-9a47-de3d85a1c2df" path="/var/lib/kubelet/pods/61e02d23-85e6-4c34-9a47-de3d85a1c2df/volumes" Jul 28 17:41:54.638178 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:54.638142 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:41:55.892914 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:55.892871 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:41:59.715051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:41:59.715012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:23.826426 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:23.826392 2579 scope.go:117] "RemoveContainer" containerID="c6ca7a017447abdf3ab277a666cc0436c3c00442c39c82df2c5755b6f5669f90" Jul 28 17:42:36.066734 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:36.066703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:37.251583 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:37.251537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:43.357195 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:43.357150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:43.495269 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:43.495232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:43.686007 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:43.685923 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg"] Jul 28 17:42:43.686166 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:43.686100 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" containerID="cri-o://92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf" gracePeriod=30 Jul 28 17:42:43.699613 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:43.699581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:43.953662 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:43.953573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:45.076098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:45.076056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:45.176195 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:45.176157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:45.276283 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:45.276247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:46.041615 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:46.041524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:46.134079 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:46.134041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:46.236814 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:46.236779 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:47.520902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:47.520866 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:42:47.643679 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:47.643650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:48.952810 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:48.952739 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:42:49.112882 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.112838 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n25n9\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-kube-api-access-n25n9\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.112882 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.112885 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-podinfo\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113168 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.112913 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-certs\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113168 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.112945 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-data\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113168 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113155 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-credential-socket\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113206 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa16f-aa81-4347-b03c-43ff52c25931-istiod-ca-cert\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-token\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113254 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:49.113339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113318 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-socket\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113373 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-envoy\") pod \"2e6fa16f-aa81-4347-b03c-43ff52c25931\" (UID: \"2e6fa16f-aa81-4347-b03c-43ff52c25931\") " Jul 28 17:42:49.113533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113406 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:49.113636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:49.113691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113642 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6fa16f-aa81-4347-b03c-43ff52c25931-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:42:49.113736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113694 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.113736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113712 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.113736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113727 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.113902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.113875 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-data" (OuterVolumeSpecName: "istio-data") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:49.115246 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.115220 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:42:49.115246 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.115225 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-kube-api-access-n25n9" (OuterVolumeSpecName: "kube-api-access-n25n9") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "kube-api-access-n25n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:42:49.115429 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.115402 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-token" (OuterVolumeSpecName: "istio-token") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:42:49.115548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.115528 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2e6fa16f-aa81-4347-b03c-43ff52c25931" (UID: "2e6fa16f-aa81-4347-b03c-43ff52c25931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:42:49.214190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.214072 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.214190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.214130 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n25n9\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-kube-api-access-n25n9\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.214190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.214141 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.214190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.214150 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.214190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.214159 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e6fa16f-aa81-4347-b03c-43ff52c25931-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.214190 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.214166 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e6fa16f-aa81-4347-b03c-43ff52c25931-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:42:49.447576 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.447540 2579 generic.go:356] "Generic (PLEG): container finished" podID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerID="92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf" exitCode=0 Jul 28 17:42:49.447755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.447588 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" Jul 28 17:42:49.447755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.447619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" event={"ID":"2e6fa16f-aa81-4347-b03c-43ff52c25931","Type":"ContainerDied","Data":"92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf"} Jul 28 17:42:49.447755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.447672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg" event={"ID":"2e6fa16f-aa81-4347-b03c-43ff52c25931","Type":"ContainerDied","Data":"308dbf9e6d5b797b835b210e6c2c91b6f6045a4c9f734c799d935fd5f5ad355e"} Jul 28 17:42:49.447755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.447698 2579 scope.go:117] "RemoveContainer" containerID="92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf" Jul 28 17:42:49.456872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.456851 2579 scope.go:117] "RemoveContainer" containerID="c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96" Jul 28 17:42:49.465481 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.465455 2579 scope.go:117] "RemoveContainer" containerID="92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf" Jul 28 17:42:49.465793 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:42:49.465771 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf\": container with ID starting with 92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf not found: ID does not exist" containerID="92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf" Jul 28 17:42:49.465852 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.465804 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf"} err="failed to get container status \"92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf\": rpc error: code = NotFound desc = could not find container \"92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf\": container with ID starting with 92c4275f3e3f4d621ef06ab838c42242ca7417ea39246cf8aec4d66db26bfadf not found: ID does not exist" Jul 28 17:42:49.465852 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.465823 2579 scope.go:117] "RemoveContainer" containerID="c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96" Jul 28 17:42:49.466077 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:42:49.466058 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96\": container with ID starting with c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96 not found: ID does not exist" containerID="c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96" Jul 28 17:42:49.466215 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.466083 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96"} err="failed to get container status \"c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96\": rpc error: code = NotFound desc = could not find container \"c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96\": container with ID starting with c785b8aa162a634a1d8511059019afd5683af1262eba7d3ad4c218d21e88de96 not found: ID does not exist" Jul 28 17:42:49.475054 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.475021 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg"] Jul 28 17:42:49.482392 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.482363 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4794d51a-openshift-default-57b4d64b4-kttfg"] Jul 28 17:42:49.684644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:49.684610 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" path="/var/lib/kubelet/pods/2e6fa16f-aa81-4347-b03c-43ff52c25931/volumes" Jul 28 17:42:54.241906 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:54.241873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:54.341296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:54.341259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:54.937812 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:54.937762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:42:57.697588 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:42:57.697552 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:43:01.918354 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:01.918314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:02.040636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:02.040604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:02.195546 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:02.195462 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:02.291692 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:02.291657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:12.729189 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:12.729150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:13.079316 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:13.079228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:24.140480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:24.140393 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:43:24.147990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:24.147966 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:43:34.103850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:34.103814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:41.733290 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:41.733252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:41.830182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:41.830094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:42.085649 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.085614 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:42.453521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453418 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf"] Jul 28 17:43:42.453858 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453843 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" Jul 28 17:43:42.453902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453861 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" Jul 28 17:43:42.453902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453874 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" Jul 28 17:43:42.453902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453883 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" Jul 28 17:43:42.454000 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453904 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61e02d23-85e6-4c34-9a47-de3d85a1c2df" containerName="revoke-keys" Jul 28 17:43:42.454000 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453913 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61e02d23-85e6-4c34-9a47-de3d85a1c2df" containerName="revoke-keys" Jul 28 17:43:42.454000 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453989 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61e02d23-85e6-4c34-9a47-de3d85a1c2df" containerName="revoke-keys" Jul 28 17:43:42.454000 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.453999 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" Jul 28 17:43:42.454139 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.454009 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e6fa16f-aa81-4347-b03c-43ff52c25931" containerName="istio-proxy" Jul 28 17:43:42.457326 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.457298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.460983 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.460960 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7dbdbf-openshift-default-dockercfg-wfpss\"" Jul 28 17:43:42.503269 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.503231 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:42.527268 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.527232 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf"] Jul 28 17:43:42.563357 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-certs\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563576 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459nv\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-kube-api-access-459nv\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563576 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563429 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-envoy\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563576 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563505 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338c0ea-3092-4053-a6cf-380eb5990785-istiod-ca-cert\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563704 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-data\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563704 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338c0ea-3092-4053-a6cf-380eb5990785-istio-podinfo\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563704 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563666 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-istio-token\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563813 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-socket\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.563813 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.563756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-credential-socket\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.597303 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.597260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:42.664455 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-data\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664455 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338c0ea-3092-4053-a6cf-380eb5990785-istio-podinfo\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-istio-token\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-socket\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-credential-socket\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-certs\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459nv\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-kube-api-access-459nv\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-envoy\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.664726 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338c0ea-3092-4053-a6cf-380eb5990785-istiod-ca-cert\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.665145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-data\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.665145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664924 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-socket\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.665145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.664944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-credential-socket\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.665145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.665029 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-certs\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.665599 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.665578 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338c0ea-3092-4053-a6cf-380eb5990785-istiod-ca-cert\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.667023 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.666998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-envoy\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.667378 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.667362 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338c0ea-3092-4053-a6cf-380eb5990785-istio-podinfo\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.674257 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.674231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-istio-token\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.675238 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.675218 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-459nv\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-kube-api-access-459nv\") pod \"e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.767805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.767707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:42.909308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.909276 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf"] Jul 28 17:43:42.911419 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:43:42.911389 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7338c0ea_3092_4053_a6cf_380eb5990785.slice/crio-b30b9a6122d15f888a44fbd650cfae353ff4ecc3a9c234877e47d2c748e9243c WatchSource:0}: Error finding container b30b9a6122d15f888a44fbd650cfae353ff4ecc3a9c234877e47d2c748e9243c: Status 404 returned error can't find the container with id b30b9a6122d15f888a44fbd650cfae353ff4ecc3a9c234877e47d2c748e9243c Jul 28 17:43:42.913525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.913491 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:42.913615 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.913563 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:42.913615 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:42.913595 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:43.722894 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:43.722856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" event={"ID":"7338c0ea-3092-4053-a6cf-380eb5990785","Type":"ContainerStarted","Data":"d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979"} Jul 28 17:43:43.723191 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:43.722909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" event={"ID":"7338c0ea-3092-4053-a6cf-380eb5990785","Type":"ContainerStarted","Data":"b30b9a6122d15f888a44fbd650cfae353ff4ecc3a9c234877e47d2c748e9243c"} Jul 28 17:43:43.748711 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:43.748654 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" podStartSLOduration=1.748617203 podStartE2EDuration="1.748617203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:43.746649967 +0000 UTC m=+2120.550338558" watchObservedRunningTime="2026-07-28 17:43:43.748617203 +0000 UTC m=+2120.552305792" Jul 28 17:43:43.768787 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:43.768747 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:43.773709 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:43.773681 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:44.727443 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:44.727411 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:44.728356 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:44.728328 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:43:45.708958 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:45.708914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:45.808365 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:45.808327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:45.950482 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:45.950453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:46.310291 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:46.310257 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:46.470030 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:46.469995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:46.568340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:46.568254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:50.213930 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:50.213896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:50.563019 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:50.562938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:50.899234 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:50.899202 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq"] Jul 28 17:43:50.902837 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:50.902813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:50.906546 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:50.906523 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8796a3-openshift-default-dockercfg-vx4xn\"" Jul 28 17:43:50.918057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:50.918026 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq"] Jul 28 17:43:50.927465 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:50.927422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:51.040529 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-data\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.040745 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsw66\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-kube-api-access-vsw66\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.040745 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-token\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.040745 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-socket\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.040895 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-credential-socket\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.040895 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-envoy\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.040895 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-podinfo\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.040895 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-certs\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.041053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.040906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340e962-5b54-4184-b49e-ecaa0bbb3189-istiod-ca-cert\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141359 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-data\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsw66\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-kube-api-access-vsw66\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-token\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-socket\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-credential-socket\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-envoy\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141527 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-podinfo\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-certs\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340e962-5b54-4184-b49e-ecaa0bbb3189-istiod-ca-cert\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-data\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.141904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.141825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-socket\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.142297 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.142261 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-credential-socket\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.142437 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.142323 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-certs\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.142717 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.142693 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340e962-5b54-4184-b49e-ecaa0bbb3189-istiod-ca-cert\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.144092 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.144069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-envoy\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.144201 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.144149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-podinfo\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.154521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.154455 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-token\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.154620 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.154524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsw66\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-kube-api-access-vsw66\") pod \"e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.198458 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.198417 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:43:51.214049 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.214016 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:51.266996 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.266950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:51.354893 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.354868 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq"] Jul 28 17:43:51.357062 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:43:51.357034 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode340e962_5b54_4184_b49e_ecaa0bbb3189.slice/crio-c2f561dc80c01e075ffe592bd60effd19dff31e88f12b270d634bf629d4e80f9 WatchSource:0}: Error finding container c2f561dc80c01e075ffe592bd60effd19dff31e88f12b270d634bf629d4e80f9: Status 404 returned error can't find the container with id c2f561dc80c01e075ffe592bd60effd19dff31e88f12b270d634bf629d4e80f9 Jul 28 17:43:51.359343 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.359310 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:51.359457 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.359396 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:51.359457 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.359445 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:51.768025 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.767914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" event={"ID":"e340e962-5b54-4184-b49e-ecaa0bbb3189","Type":"ContainerStarted","Data":"d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879"} Jul 28 17:43:51.768025 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.767975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" event={"ID":"e340e962-5b54-4184-b49e-ecaa0bbb3189","Type":"ContainerStarted","Data":"c2f561dc80c01e075ffe592bd60effd19dff31e88f12b270d634bf629d4e80f9"} Jul 28 17:43:51.792817 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:51.792757 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podStartSLOduration=1.7927356319999999 podStartE2EDuration="1.792735632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:43:51.791439746 +0000 UTC m=+2128.595128337" watchObservedRunningTime="2026-07-28 17:43:51.792735632 +0000 UTC m=+2128.596424223" Jul 28 17:43:52.214968 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:52.214922 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:52.216298 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:52.216270 2579 patch_prober.go:28] interesting pod/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 28 17:43:52.216430 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:52.216322 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 28 17:43:53.218343 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:53.218310 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:53.777843 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:53.777801 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:53.778691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:53.778673 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:43:53.840131 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:53.839912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:53.946749 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:53.946707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:54.141939 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:54.141895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:54.614129 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:54.614079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:54.709413 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:54.709382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:54.791880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:54.791841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:58.282474 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:58.282439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:43:59.714592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.714557 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:43:59.729210 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.729177 2579 patch_prober.go:28] interesting pod/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 28 17:43:59.729350 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.729228 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 28 17:43:59.813410 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.813379 2579 generic.go:356] "Generic (PLEG): container finished" podID="7338c0ea-3092-4053-a6cf-380eb5990785" containerID="d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979" exitCode=0 Jul 28 17:43:59.813598 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.813452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" event={"ID":"7338c0ea-3092-4053-a6cf-380eb5990785","Type":"ContainerDied","Data":"d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979"} Jul 28 17:43:59.813875 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.813858 2579 scope.go:117] "RemoveContainer" containerID="d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979" Jul 28 17:43:59.814819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.814790 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:59.814913 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.814887 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:43:59.814976 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:43:59.814945 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:44:00.820656 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:00.820618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" event={"ID":"7338c0ea-3092-4053-a6cf-380eb5990785","Type":"ContainerStarted","Data":"26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e"} Jul 28 17:44:01.768592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:01.768551 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:44:01.769010 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:01.768980 2579 patch_prober.go:28] interesting pod/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 28 17:44:01.769254 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:01.769228 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 28 17:44:02.769070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:02.769016 2579 patch_prober.go:28] interesting pod/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 28 17:44:02.769632 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:02.769090 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 28 17:44:03.768629 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:03.768587 2579 patch_prober.go:28] interesting pod/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 28 17:44:03.768950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:03.768919 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 28 17:44:04.771977 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:04.771946 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:44:04.772399 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:04.772226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:44:04.772900 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:04.772884 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:44:05.348721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:05.348688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:44:05.445069 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:05.445029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:44:08.779460 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.779420 2579 patch_prober.go:28] interesting pod/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 28 17:44:08.779879 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.779488 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 28 17:44:08.873713 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.873681 2579 generic.go:356] "Generic (PLEG): container finished" podID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerID="d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879" exitCode=0 Jul 28 17:44:08.873713 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.873721 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" event={"ID":"e340e962-5b54-4184-b49e-ecaa0bbb3189","Type":"ContainerDied","Data":"d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879"} Jul 28 17:44:08.874083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.874071 2579 scope.go:117] "RemoveContainer" containerID="d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879" Jul 28 17:44:08.875098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.875067 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:44:08.875240 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.875169 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:44:08.875240 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:08.875218 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:44:09.880078 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:09.880045 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" event={"ID":"e340e962-5b54-4184-b49e-ecaa0bbb3189","Type":"ContainerStarted","Data":"fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718"} Jul 28 17:44:10.214931 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:10.214830 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:44:10.215141 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:10.215091 2579 patch_prober.go:28] interesting pod/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 28 17:44:10.215195 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:10.215167 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 28 17:44:11.214744 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:11.214703 2579 patch_prober.go:28] interesting pod/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 28 17:44:11.215267 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:11.214797 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 28 17:44:12.213711 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:12.213678 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:44:12.214698 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:12.214675 2579 patch_prober.go:28] interesting pod/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 28 17:44:12.214756 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:12.214726 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 28 17:44:13.215381 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:13.215340 2579 patch_prober.go:28] interesting pod/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 28 17:44:13.215803 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:13.215401 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 28 17:44:14.218817 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:14.218790 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:44:14.219284 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:14.219085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:44:14.219819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:14.219797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:44:15.705236 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:15.705151 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:44:23.857839 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:23.857797 2579 scope.go:117] "RemoveContainer" containerID="2034733444ff94d5f0cf98f18b5cd2fecf4a20333674975cf9b58dd1910ca1f1" Jul 28 17:44:27.510699 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:27.510657 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:44:29.007430 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.007390 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85"] Jul 28 17:44:29.010421 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.010403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.014957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.014934 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8796a3\"/\"kube-root-ca.crt\"" Jul 28 17:44:29.015126 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.014933 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8796a3\"/\"test-model-8796a3-kserve-self-signed-certs\"" Jul 28 17:44:29.015126 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.014933 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8796a3\"/\"openshift-service-ca.crt\"" Jul 28 17:44:29.015126 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.014934 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8796a3\"/\"default-dockercfg-xs886\"" Jul 28 17:44:29.021779 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.021756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85"] Jul 28 17:44:29.078082 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.078047 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-dshm\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.078082 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.078085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-tmp-dir\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.078344 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.078189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-model-cache\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.078344 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.078229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-home\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.078344 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.078250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d130a6-0289-49ce-acd5-6379f6d5394d-tls-certs\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.078344 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.078298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zj9w\" (UniqueName: \"kubernetes.io/projected/61d130a6-0289-49ce-acd5-6379f6d5394d-kube-api-access-9zj9w\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.099915 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.099880 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:44:29.179575 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.179541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zj9w\" (UniqueName: \"kubernetes.io/projected/61d130a6-0289-49ce-acd5-6379f6d5394d-kube-api-access-9zj9w\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.179780 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.179599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-dshm\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.179780 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.179623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-tmp-dir\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.179780 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.179659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-model-cache\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.179780 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.179684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-home\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.179780 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.179702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d130a6-0289-49ce-acd5-6379f6d5394d-tls-certs\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.180173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.180098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-model-cache\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.180301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.180138 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-home\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.180301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.180182 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-tmp-dir\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.181911 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.181891 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-dshm\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.182233 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.182215 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d130a6-0289-49ce-acd5-6379f6d5394d-tls-certs\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.188590 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.188568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zj9w\" (UniqueName: \"kubernetes.io/projected/61d130a6-0289-49ce-acd5-6379f6d5394d-kube-api-access-9zj9w\") pod \"test-model-8796a3-kserve-67f754bddf-v9d85\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.319739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.319637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:29.452559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.452525 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85"] Jul 28 17:44:29.454194 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:44:29.454160 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d130a6_0289_49ce_acd5_6379f6d5394d.slice/crio-ccb673b967fa63fefca274f1b5d85ac82e8c8f4e8cd5a8b269e09f141b8291af WatchSource:0}: Error finding container ccb673b967fa63fefca274f1b5d85ac82e8c8f4e8cd5a8b269e09f141b8291af: Status 404 returned error can't find the container with id ccb673b967fa63fefca274f1b5d85ac82e8c8f4e8cd5a8b269e09f141b8291af Jul 28 17:44:29.976924 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.976890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" event={"ID":"61d130a6-0289-49ce-acd5-6379f6d5394d","Type":"ContainerStarted","Data":"ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683"} Jul 28 17:44:29.977160 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.976936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" event={"ID":"61d130a6-0289-49ce-acd5-6379f6d5394d","Type":"ContainerStarted","Data":"ccb673b967fa63fefca274f1b5d85ac82e8c8f4e8cd5a8b269e09f141b8291af"} Jul 28 17:44:29.997403 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:29.997347 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" podStartSLOduration=1.997328728 podStartE2EDuration="1.997328728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:29.996376606 +0000 UTC m=+2166.800065209" watchObservedRunningTime="2026-07-28 17:44:29.997328728 +0000 UTC m=+2166.801017359" Jul 28 17:44:31.899779 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:31.899734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:44:39.320472 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:39.320422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:39.320968 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:39.320496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:39.333195 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:39.333163 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:40.037782 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:40.037750 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:44:46.604733 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:44:46.604692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:45:07.910090 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:07.910055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:08.854161 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:08.854123 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85"] Jul 28 17:45:08.879437 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:08.879399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:10.309774 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:10.309732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:45:19.498038 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:19.497997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:19.576636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:19.576590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:19.639028 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:19.638983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:22.301585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:22.301549 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:45:23.873392 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:23.873346 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" podUID="61d130a6-0289-49ce-acd5-6379f6d5394d" containerName="main" containerID="cri-o://ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683" gracePeriod=45 Jul 28 17:45:24.074371 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.074334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:24.127429 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.127345 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:45:24.249217 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.249181 2579 generic.go:356] "Generic (PLEG): container finished" podID="61d130a6-0289-49ce-acd5-6379f6d5394d" containerID="ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683" exitCode=0 Jul 28 17:45:24.249217 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.249220 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" Jul 28 17:45:24.249456 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.249271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" event={"ID":"61d130a6-0289-49ce-acd5-6379f6d5394d","Type":"ContainerDied","Data":"ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683"} Jul 28 17:45:24.249456 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.249307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85" event={"ID":"61d130a6-0289-49ce-acd5-6379f6d5394d","Type":"ContainerDied","Data":"ccb673b967fa63fefca274f1b5d85ac82e8c8f4e8cd5a8b269e09f141b8291af"} Jul 28 17:45:24.249456 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.249325 2579 scope.go:117] "RemoveContainer" containerID="ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683" Jul 28 17:45:24.259983 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.259957 2579 scope.go:117] "RemoveContainer" containerID="ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683" Jul 28 17:45:24.260361 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:45:24.260339 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683\": container with ID starting with ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683 not found: ID does not exist" containerID="ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683" Jul 28 17:45:24.260426 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.260373 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683"} err="failed to get container status \"ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683\": rpc error: code = NotFound desc = could not find container \"ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683\": container with ID starting with ead34558ce5e534d6a07c3008cc8bf8807c80d1b231ef8b692dc3c5c3e5c1683 not found: ID does not exist" Jul 28 17:45:24.275691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.275661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zj9w\" (UniqueName: \"kubernetes.io/projected/61d130a6-0289-49ce-acd5-6379f6d5394d-kube-api-access-9zj9w\") pod \"61d130a6-0289-49ce-acd5-6379f6d5394d\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " Jul 28 17:45:24.275786 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.275725 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-model-cache\") pod \"61d130a6-0289-49ce-acd5-6379f6d5394d\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " Jul 28 17:45:24.275786 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.275772 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-dshm\") pod \"61d130a6-0289-49ce-acd5-6379f6d5394d\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " Jul 28 17:45:24.275912 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.275793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-home\") pod \"61d130a6-0289-49ce-acd5-6379f6d5394d\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " Jul 28 17:45:24.275912 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.275821 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-tmp-dir\") pod \"61d130a6-0289-49ce-acd5-6379f6d5394d\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " Jul 28 17:45:24.275912 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.275842 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d130a6-0289-49ce-acd5-6379f6d5394d-tls-certs\") pod \"61d130a6-0289-49ce-acd5-6379f6d5394d\" (UID: \"61d130a6-0289-49ce-acd5-6379f6d5394d\") " Jul 28 17:45:24.276086 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.276058 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61d130a6-0289-49ce-acd5-6379f6d5394d" (UID: "61d130a6-0289-49ce-acd5-6379f6d5394d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:24.276170 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.276073 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-model-cache" (OuterVolumeSpecName: "model-cache") pod "61d130a6-0289-49ce-acd5-6379f6d5394d" (UID: "61d130a6-0289-49ce-acd5-6379f6d5394d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:24.276170 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.276081 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-home" (OuterVolumeSpecName: "home") pod "61d130a6-0289-49ce-acd5-6379f6d5394d" (UID: "61d130a6-0289-49ce-acd5-6379f6d5394d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:24.278153 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.278125 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d130a6-0289-49ce-acd5-6379f6d5394d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61d130a6-0289-49ce-acd5-6379f6d5394d" (UID: "61d130a6-0289-49ce-acd5-6379f6d5394d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:45:24.278153 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.278140 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-dshm" (OuterVolumeSpecName: "dshm") pod "61d130a6-0289-49ce-acd5-6379f6d5394d" (UID: "61d130a6-0289-49ce-acd5-6379f6d5394d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:24.278306 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.278200 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d130a6-0289-49ce-acd5-6379f6d5394d-kube-api-access-9zj9w" (OuterVolumeSpecName: "kube-api-access-9zj9w") pod "61d130a6-0289-49ce-acd5-6379f6d5394d" (UID: "61d130a6-0289-49ce-acd5-6379f6d5394d"). InnerVolumeSpecName "kube-api-access-9zj9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:45:24.377534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.377434 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-dshm\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:24.377534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.377468 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-home\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:24.377534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.377477 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-tmp-dir\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:24.377534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.377487 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d130a6-0289-49ce-acd5-6379f6d5394d-tls-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:24.377534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.377496 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zj9w\" (UniqueName: \"kubernetes.io/projected/61d130a6-0289-49ce-acd5-6379f6d5394d-kube-api-access-9zj9w\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:24.377534 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.377504 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d130a6-0289-49ce-acd5-6379f6d5394d-model-cache\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:24.514699 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.514660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:24.578889 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.578850 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85"] Jul 28 17:45:24.583746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:24.583710 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8796a3/test-model-8796a3-kserve-67f754bddf-v9d85"] Jul 28 17:45:25.684843 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:25.684813 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61d130a6-0289-49ce-acd5-6379f6d5394d" path="/var/lib/kubelet/pods/61d130a6-0289-49ce-acd5-6379f6d5394d/volumes" Jul 28 17:45:30.999458 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:30.999423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:31.069892 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.069794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:31.288790 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.288693 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf"] Jul 28 17:45:31.288973 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.288913 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" containerID="cri-o://26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e" gracePeriod=30 Jul 28 17:45:31.316215 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.316091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:31.534369 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.534335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:31.664654 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.664620 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg"] Jul 28 17:45:31.664974 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.664961 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d130a6-0289-49ce-acd5-6379f6d5394d" containerName="main" Jul 28 17:45:31.665032 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.664975 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d130a6-0289-49ce-acd5-6379f6d5394d" containerName="main" Jul 28 17:45:31.665072 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.665046 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d130a6-0289-49ce-acd5-6379f6d5394d" containerName="main" Jul 28 17:45:31.667864 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.667846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" Jul 28 17:45:31.671116 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.671082 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 17:45:31.671207 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.671083 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 17:45:31.685820 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.685791 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg"] Jul 28 17:45:31.745682 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.745645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21949a02-464b-405d-be93-ce83381e1e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg\" (UID: \"21949a02-464b-405d-be93-ce83381e1e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" Jul 28 17:45:31.846091 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.846041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21949a02-464b-405d-be93-ce83381e1e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg\" (UID: \"21949a02-464b-405d-be93-ce83381e1e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" Jul 28 17:45:31.846767 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.846745 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21949a02-464b-405d-be93-ce83381e1e5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg\" (UID: \"21949a02-464b-405d-be93-ce83381e1e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" Jul 28 17:45:31.978624 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:31.978518 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" Jul 28 17:45:32.121113 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:32.121062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg"] Jul 28 17:45:32.122584 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:45:32.122557 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21949a02_464b_405d_be93_ce83381e1e5a.slice/crio-eb3e9e2e0dbfd9d01aa99a30ae7ff36e1a8eed8985d0a0e919b8865326e22b16 WatchSource:0}: Error finding container eb3e9e2e0dbfd9d01aa99a30ae7ff36e1a8eed8985d0a0e919b8865326e22b16: Status 404 returned error can't find the container with id eb3e9e2e0dbfd9d01aa99a30ae7ff36e1a8eed8985d0a0e919b8865326e22b16 Jul 28 17:45:32.292745 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:32.292653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" event={"ID":"21949a02-464b-405d-be93-ce83381e1e5a","Type":"ContainerStarted","Data":"eb3e9e2e0dbfd9d01aa99a30ae7ff36e1a8eed8985d0a0e919b8865326e22b16"} Jul 28 17:45:33.299186 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:33.299149 2579 generic.go:356] "Generic (PLEG): container finished" podID="21949a02-464b-405d-be93-ce83381e1e5a" containerID="4082981fd7ce7f372ff398901ea28b6060af4e8f18893faf24e60e7ba2adec2d" exitCode=0 Jul 28 17:45:33.299593 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:33.299235 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" event={"ID":"21949a02-464b-405d-be93-ce83381e1e5a","Type":"ContainerDied","Data":"4082981fd7ce7f372ff398901ea28b6060af4e8f18893faf24e60e7ba2adec2d"} Jul 28 17:45:34.433415 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:34.433381 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" Jul 28 17:45:34.470217 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:34.470185 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21949a02-464b-405d-be93-ce83381e1e5a-maas-api-service-ca\") pod \"21949a02-464b-405d-be93-ce83381e1e5a\" (UID: \"21949a02-464b-405d-be93-ce83381e1e5a\") " Jul 28 17:45:34.470616 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:34.470591 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21949a02-464b-405d-be93-ce83381e1e5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "21949a02-464b-405d-be93-ce83381e1e5a" (UID: "21949a02-464b-405d-be93-ce83381e1e5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:45:34.571028 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:34.570926 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21949a02-464b-405d-be93-ce83381e1e5a-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:34.774286 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:34.774254 2579 patch_prober.go:28] interesting pod/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 17:45:34.774451 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:34.774311 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 17:45:35.311154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:35.311097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" event={"ID":"21949a02-464b-405d-be93-ce83381e1e5a","Type":"ContainerDied","Data":"eb3e9e2e0dbfd9d01aa99a30ae7ff36e1a8eed8985d0a0e919b8865326e22b16"} Jul 28 17:45:35.311154 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:35.311156 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg" Jul 28 17:45:35.311377 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:35.311179 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb3e9e2e0dbfd9d01aa99a30ae7ff36e1a8eed8985d0a0e919b8865326e22b16" Jul 28 17:45:36.530215 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.530180 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:45:36.590988 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.590953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338c0ea-3092-4053-a6cf-380eb5990785-istio-podinfo\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591001 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-data\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591041 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-socket\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338c0ea-3092-4053-a6cf-380eb5990785-istiod-ca-cert\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591146 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-envoy\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591243 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591205 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-459nv\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-kube-api-access-459nv\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-certs\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591273 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-istio-token\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591297 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-credential-socket\") pod \"7338c0ea-3092-4053-a6cf-380eb5990785\" (UID: \"7338c0ea-3092-4053-a6cf-380eb5990785\") " Jul 28 17:45:36.591513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591450 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:36.591513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591493 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7338c0ea-3092-4053-a6cf-380eb5990785-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:45:36.591755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591641 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7338c0ea-3092-4053-a6cf-380eb5990785-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.591755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591662 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.591755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591644 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:36.591755 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591679 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:36.591929 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.591909 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-data" (OuterVolumeSpecName: "istio-data") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:36.593388 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.593363 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7338c0ea-3092-4053-a6cf-380eb5990785-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:45:36.593602 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.593587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-istio-token" (OuterVolumeSpecName: "istio-token") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:45:36.593790 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.593758 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-kube-api-access-459nv" (OuterVolumeSpecName: "kube-api-access-459nv") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "kube-api-access-459nv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:45:36.593790 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.593768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7338c0ea-3092-4053-a6cf-380eb5990785" (UID: "7338c0ea-3092-4053-a6cf-380eb5990785"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:36.692449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.692397 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-459nv\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-kube-api-access-459nv\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.692449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.692445 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.692449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.692456 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7338c0ea-3092-4053-a6cf-380eb5990785-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.692449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.692465 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.692746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.692474 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7338c0ea-3092-4053-a6cf-380eb5990785-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.692746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.692483 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:36.692746 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:36.692491 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7338c0ea-3092-4053-a6cf-380eb5990785-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:37.323495 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.323461 2579 generic.go:356] "Generic (PLEG): container finished" podID="7338c0ea-3092-4053-a6cf-380eb5990785" containerID="26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e" exitCode=0 Jul 28 17:45:37.323666 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.323507 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" Jul 28 17:45:37.323666 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.323555 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" event={"ID":"7338c0ea-3092-4053-a6cf-380eb5990785","Type":"ContainerDied","Data":"26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e"} Jul 28 17:45:37.323666 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.323619 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf" event={"ID":"7338c0ea-3092-4053-a6cf-380eb5990785","Type":"ContainerDied","Data":"b30b9a6122d15f888a44fbd650cfae353ff4ecc3a9c234877e47d2c748e9243c"} Jul 28 17:45:37.323666 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.323643 2579 scope.go:117] "RemoveContainer" containerID="26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e" Jul 28 17:45:37.333697 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.333671 2579 scope.go:117] "RemoveContainer" containerID="d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979" Jul 28 17:45:37.342223 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.342202 2579 scope.go:117] "RemoveContainer" containerID="26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e" Jul 28 17:45:37.342740 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:45:37.342719 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e\": container with ID starting with 26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e not found: ID does not exist" containerID="26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e" Jul 28 17:45:37.342797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.342751 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e"} err="failed to get container status \"26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e\": rpc error: code = NotFound desc = could not find container \"26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e\": container with ID starting with 26087d8bc6750595536f140c23767a20f1a8cb0911c5df210fb42412a3ca3c3e not found: ID does not exist" Jul 28 17:45:37.342797 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.342770 2579 scope.go:117] "RemoveContainer" containerID="d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979" Jul 28 17:45:37.343029 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:45:37.343012 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979\": container with ID starting with d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979 not found: ID does not exist" containerID="d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979" Jul 28 17:45:37.343079 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.343035 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979"} err="failed to get container status \"d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979\": rpc error: code = NotFound desc = could not find container \"d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979\": container with ID starting with d6ff59be1e786fc52137e030751bb5eb8e4286bd9c02fbfa99729ed8d6df7979 not found: ID does not exist" Jul 28 17:45:37.358067 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.358038 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf"] Jul 28 17:45:37.363242 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.363213 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7dbdbf-openshift-default-77dbf84f56-hvxnf"] Jul 28 17:45:37.684252 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:37.684218 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" path="/var/lib/kubelet/pods/7338c0ea-3092-4053-a6cf-380eb5990785/volumes" Jul 28 17:45:42.384841 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:42.384807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:42.471340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:42.471305 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:42.538283 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:42.538254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:45.505426 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:45.505337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:45:46.748809 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:46.748776 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg"] Jul 28 17:45:46.752675 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:46.752644 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8796a3-lzplg"] Jul 28 17:45:46.771339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:46.771312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:47.148336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:47.148302 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:47.684973 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:47.684938 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21949a02-464b-405d-be93-ce83381e1e5a" path="/var/lib/kubelet/pods/21949a02-464b-405d-be93-ce83381e1e5a/volumes" Jul 28 17:45:53.532263 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:53.532226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:53.593634 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:53.593601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:53.805452 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:53.805355 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq"] Jul 28 17:45:53.805630 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:53.805569 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" containerID="cri-o://fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718" gracePeriod=30 Jul 28 17:45:53.826309 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:53.826272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:53.925831 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:53.925799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:54.077548 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.077458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:54.442217 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.442182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:54.766904 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.766826 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd"] Jul 28 17:45:54.767305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767186 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21949a02-464b-405d-be93-ce83381e1e5a" containerName="revoke-keys" Jul 28 17:45:54.767305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767199 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21949a02-464b-405d-be93-ce83381e1e5a" containerName="revoke-keys" Jul 28 17:45:54.767305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767211 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" Jul 28 17:45:54.767305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767218 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" Jul 28 17:45:54.767305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767225 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" Jul 28 17:45:54.767305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767234 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" Jul 28 17:45:54.767614 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767313 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" Jul 28 17:45:54.767614 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767322 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7338c0ea-3092-4053-a6cf-380eb5990785" containerName="istio-proxy" Jul 28 17:45:54.767614 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.767328 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21949a02-464b-405d-be93-ce83381e1e5a" containerName="revoke-keys" Jul 28 17:45:54.771957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.771935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.775181 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.775160 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7869f9-openshift-default-dockercfg-rlwdg\"" Jul 28 17:45:54.787730 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.787695 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd"] Jul 28 17:45:54.800787 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.800755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:54.847511 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-podinfo\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-token\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-credential-socket\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-certs\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-data\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7np7z\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-kube-api-access-7np7z\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847983 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847774 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-socket\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847983 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-envoy\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.847983 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.847865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b75f6a0-8763-4e84-b7bd-83a12633c913-istiod-ca-cert\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.907885 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.907848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:54.948343 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-token\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-credential-socket\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-certs\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-data\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7np7z\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-kube-api-access-7np7z\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-socket\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-envoy\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948920 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b75f6a0-8763-4e84-b7bd-83a12633c913-istiod-ca-cert\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948920 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-podinfo\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.948920 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-certs\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.949101 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.948937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-data\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.949101 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.949130 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-credential-socket\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.949256 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.949231 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-socket\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.949620 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.949600 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b75f6a0-8763-4e84-b7bd-83a12633c913-istiod-ca-cert\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.951440 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.951416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-envoy\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.951647 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.951632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-podinfo\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.959728 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.959701 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-token\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:54.961014 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:54.960992 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7np7z\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-kube-api-access-7np7z\") pod \"e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:55.083022 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.082932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:55.232516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.232486 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd"] Jul 28 17:45:55.235196 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:45:55.235166 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b75f6a0_8763_4e84_b7bd_83a12633c913.slice/crio-47d8e0c4469458dbc60d13af72b51ed05376ee48b47037219b1c6b25fe27eeef WatchSource:0}: Error finding container 47d8e0c4469458dbc60d13af72b51ed05376ee48b47037219b1c6b25fe27eeef: Status 404 returned error can't find the container with id 47d8e0c4469458dbc60d13af72b51ed05376ee48b47037219b1c6b25fe27eeef Jul 28 17:45:55.237424 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.237392 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:45:55.237545 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.237479 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:45:55.237545 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.237528 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:45:55.416899 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.416861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" event={"ID":"2b75f6a0-8763-4e84-b7bd-83a12633c913","Type":"ContainerStarted","Data":"18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4"} Jul 28 17:45:55.417093 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.416915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" event={"ID":"2b75f6a0-8763-4e84-b7bd-83a12633c913","Type":"ContainerStarted","Data":"47d8e0c4469458dbc60d13af72b51ed05376ee48b47037219b1c6b25fe27eeef"} Jul 28 17:45:55.441160 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:55.441069 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" podStartSLOduration=1.441045175 podStartE2EDuration="1.441045175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:55.440160517 +0000 UTC m=+2252.243849108" watchObservedRunningTime="2026-07-28 17:45:55.441045175 +0000 UTC m=+2252.244733767" Jul 28 17:45:56.084000 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:56.083959 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:56.088944 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:56.088915 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:56.422253 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:56.422217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:56.423124 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:56.423094 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:45:57.453420 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:57.453384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:57.538570 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:57.538537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:57.637449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:57.637411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:57.737233 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:57.737128 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:58.348882 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:58.348850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:58.449927 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:58.449891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:45:59.068056 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.067969 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:45:59.188346 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188304 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-credential-socket\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188346 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188348 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-token\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-data\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188418 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340e962-5b54-4184-b49e-ecaa0bbb3189-istiod-ca-cert\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188460 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-certs\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188486 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-envoy\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-podinfo\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsw66\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-kube-api-access-vsw66\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.188617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188580 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-socket\") pod \"e340e962-5b54-4184-b49e-ecaa0bbb3189\" (UID: \"e340e962-5b54-4184-b49e-ecaa0bbb3189\") " Jul 28 17:45:59.189057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188681 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:59.189057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188737 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:59.189057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:59.189057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188900 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.189057 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.188922 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.189379 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.189153 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e340e962-5b54-4184-b49e-ecaa0bbb3189-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:45:59.189379 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.189306 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-data" (OuterVolumeSpecName: "istio-data") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:59.190826 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.190796 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-kube-api-access-vsw66" (OuterVolumeSpecName: "kube-api-access-vsw66") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "kube-api-access-vsw66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:45:59.190948 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.190866 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:45:59.190948 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.190894 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:45:59.190948 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.190931 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-token" (OuterVolumeSpecName: "istio-token") pod "e340e962-5b54-4184-b49e-ecaa0bbb3189" (UID: "e340e962-5b54-4184-b49e-ecaa0bbb3189"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:45:59.290245 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.290201 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.290245 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.290237 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.290245 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.290247 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsw66\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-kube-api-access-vsw66\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.290494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.290257 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.290494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.290268 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.290494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.290281 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e340e962-5b54-4184-b49e-ecaa0bbb3189-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.290494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.290292 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e340e962-5b54-4184-b49e-ecaa0bbb3189-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:45:59.439226 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.439194 2579 generic.go:356] "Generic (PLEG): container finished" podID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerID="fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718" exitCode=0 Jul 28 17:45:59.439393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.439244 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" Jul 28 17:45:59.439393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.439287 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" event={"ID":"e340e962-5b54-4184-b49e-ecaa0bbb3189","Type":"ContainerDied","Data":"fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718"} Jul 28 17:45:59.439393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.439348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq" event={"ID":"e340e962-5b54-4184-b49e-ecaa0bbb3189","Type":"ContainerDied","Data":"c2f561dc80c01e075ffe592bd60effd19dff31e88f12b270d634bf629d4e80f9"} Jul 28 17:45:59.439393 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.439373 2579 scope.go:117] "RemoveContainer" containerID="fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718" Jul 28 17:45:59.448522 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.448501 2579 scope.go:117] "RemoveContainer" containerID="d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879" Jul 28 17:45:59.456533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.456511 2579 scope.go:117] "RemoveContainer" containerID="fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718" Jul 28 17:45:59.456795 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:45:59.456777 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718\": container with ID starting with fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718 not found: ID does not exist" containerID="fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718" Jul 28 17:45:59.456852 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.456803 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718"} err="failed to get container status \"fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718\": rpc error: code = NotFound desc = could not find container \"fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718\": container with ID starting with fcb12cbda93018ca99f459276540cbbc493cda5ef2695cc18be5e667e3a9f718 not found: ID does not exist" Jul 28 17:45:59.456852 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.456820 2579 scope.go:117] "RemoveContainer" containerID="d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879" Jul 28 17:45:59.457095 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:45:59.457077 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879\": container with ID starting with d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879 not found: ID does not exist" containerID="d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879" Jul 28 17:45:59.457157 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.457101 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879"} err="failed to get container status \"d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879\": rpc error: code = NotFound desc = could not find container \"d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879\": container with ID starting with d58f6e98856c3235b10fcbdfbbb39c1a50999a58d089d528b7ec9044ad756879 not found: ID does not exist" Jul 28 17:45:59.472378 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.468887 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq"] Jul 28 17:45:59.475030 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.474995 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8796a3-openshift-default-dfcbbc49b-ngkfq"] Jul 28 17:45:59.685358 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:45:59.685323 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" path="/var/lib/kubelet/pods/e340e962-5b54-4184-b49e-ecaa0bbb3189/volumes" Jul 28 17:46:01.201302 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:01.201260 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:46:01.964550 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:01.964517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:02.514399 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.514359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:02.841259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.835783 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925"] Jul 28 17:46:02.841259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.836305 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" Jul 28 17:46:02.841259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.836325 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" Jul 28 17:46:02.841259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.836361 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" Jul 28 17:46:02.841259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.836371 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" Jul 28 17:46:02.841259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.836446 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" Jul 28 17:46:02.841694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.841447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.845592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.845360 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-29d9e9-openshift-default-dockercfg-trlrm\"" Jul 28 17:46:02.857389 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.857355 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925"] Jul 28 17:46:02.863826 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.863734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:02.921711 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.921670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9465f-7520-46a4-a81e-4a245b9ad423-istiod-ca-cert\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.921886 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.921722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wjc\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-kube-api-access-d4wjc\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.921886 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.921830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-certs\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.921990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.921905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-credential-socket\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.921990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.921939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-token\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.921990 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.921985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-data\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.922096 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.922053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-socket\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.922096 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.922087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-podinfo\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:02.922222 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:02.922150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-envoy\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023148 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-data\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-socket\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-podinfo\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-envoy\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9465f-7520-46a4-a81e-4a245b9ad423-istiod-ca-cert\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023325 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wjc\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-kube-api-access-d4wjc\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-certs\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023712 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-credential-socket\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.023712 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.023439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-token\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.024229 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.024201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-data\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.024506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.024484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-socket\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.028133 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.026046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9465f-7520-46a4-a81e-4a245b9ad423-istiod-ca-cert\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.028133 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.026324 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-certs\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.028133 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.026471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-credential-socket\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.029213 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.029156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-envoy\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.031213 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.031184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-podinfo\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.037776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.037734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-token\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.038850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.038814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wjc\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-kube-api-access-d4wjc\") pod \"e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.114407 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.114322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:03.163644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.163607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:03.348682 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.348645 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925"] Jul 28 17:46:03.349359 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:46:03.349327 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49a9465f_7520_46a4_a81e_4a245b9ad423.slice/crio-9adfb97ebe7dde9cb8def0b15e63c15a0b6ee2aa37e8f5a28553f2a85dc49067 WatchSource:0}: Error finding container 9adfb97ebe7dde9cb8def0b15e63c15a0b6ee2aa37e8f5a28553f2a85dc49067: Status 404 returned error can't find the container with id 9adfb97ebe7dde9cb8def0b15e63c15a0b6ee2aa37e8f5a28553f2a85dc49067 Jul 28 17:46:03.351648 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.351609 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:03.351743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.351683 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:03.351743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.351713 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:03.461845 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.461805 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" event={"ID":"49a9465f-7520-46a4-a81e-4a245b9ad423","Type":"ContainerStarted","Data":"f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179"} Jul 28 17:46:03.462039 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.461871 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" event={"ID":"49a9465f-7520-46a4-a81e-4a245b9ad423","Type":"ContainerStarted","Data":"9adfb97ebe7dde9cb8def0b15e63c15a0b6ee2aa37e8f5a28553f2a85dc49067"} Jul 28 17:46:03.492128 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:03.492064 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" podStartSLOduration=1.492048364 podStartE2EDuration="1.492048364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:03.490430841 +0000 UTC m=+2260.294119453" watchObservedRunningTime="2026-07-28 17:46:03.492048364 +0000 UTC m=+2260.295736954" Jul 28 17:46:04.164053 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:04.164021 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:04.168958 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:04.168930 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:04.469391 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:04.469291 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:04.472650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:04.472620 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:05.494362 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:05.494322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:05.600902 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:05.600866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:05.998827 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:05.998793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:06.429852 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:06.429819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:06.525842 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:06.525813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:09.496858 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:09.496768 2579 generic.go:356] "Generic (PLEG): container finished" podID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerID="18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4" exitCode=0 Jul 28 17:46:09.496858 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:09.496833 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" event={"ID":"2b75f6a0-8763-4e84-b7bd-83a12633c913","Type":"ContainerDied","Data":"18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4"} Jul 28 17:46:09.497375 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:09.497288 2579 scope.go:117] "RemoveContainer" containerID="18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4" Jul 28 17:46:09.498339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:09.498308 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:09.498473 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:09.498388 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:09.498473 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:09.498457 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:10.045301 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:10.045268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:10.517070 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:10.517036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" event={"ID":"2b75f6a0-8763-4e84-b7bd-83a12633c913","Type":"ContainerStarted","Data":"bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7"} Jul 28 17:46:11.083374 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:11.083262 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:46:11.084294 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:11.084275 2579 patch_prober.go:28] interesting pod/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 28 17:46:11.084361 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:11.084321 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 28 17:46:11.422267 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:11.422211 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:46:12.083979 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:12.083932 2579 patch_prober.go:28] interesting pod/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 28 17:46:12.084561 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:12.084046 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 28 17:46:13.084226 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:13.084187 2579 patch_prober.go:28] interesting pod/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 28 17:46:13.084642 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:13.084243 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 28 17:46:14.083336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:14.083295 2579 patch_prober.go:28] interesting pod/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 28 17:46:14.083532 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:14.083354 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 28 17:46:15.089405 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:15.089364 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:46:15.090646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:15.090622 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:46:17.002467 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:17.002430 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:46:17.563117 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:17.563071 2579 generic.go:356] "Generic (PLEG): container finished" podID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerID="f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179" exitCode=0 Jul 28 17:46:17.563304 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:17.563151 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" event={"ID":"49a9465f-7520-46a4-a81e-4a245b9ad423","Type":"ContainerDied","Data":"f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179"} Jul 28 17:46:17.563555 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:17.563529 2579 scope.go:117] "RemoveContainer" containerID="f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179" Jul 28 17:46:17.564741 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:17.564710 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:17.564816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:17.564797 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:17.564861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:17.564846 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:46:18.569559 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:18.569523 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" event={"ID":"49a9465f-7520-46a4-a81e-4a245b9ad423","Type":"ContainerStarted","Data":"61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509"} Jul 28 17:46:19.164340 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:19.164296 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:19.165470 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:19.165437 2579 patch_prober.go:28] interesting pod/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 17:46:19.165607 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:19.165499 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 17:46:19.469326 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:19.469223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:20.164861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:20.164825 2579 patch_prober.go:28] interesting pod/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 17:46:20.165321 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:20.164885 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 17:46:21.165127 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:21.165061 2579 patch_prober.go:28] interesting pod/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 17:46:21.165580 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:21.165487 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 17:46:22.165043 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:22.164994 2579 patch_prober.go:28] interesting pod/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Jul 28 17:46:22.165263 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:22.165074 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Jul 28 17:46:23.168976 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:23.168944 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:23.170083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:23.170063 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:46:33.844449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:33.844412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:33.924297 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:33.924260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:34.024547 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:34.024511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:36.311560 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:36.311520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:46:38.437925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:38.437891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:38.872626 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:38.872585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:45.610148 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:45.610093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:45.724502 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:45.724458 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:45.879881 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:45.879791 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd"] Jul 28 17:46:45.880048 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:45.880022 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" containerID="cri-o://bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7" gracePeriod=30 Jul 28 17:46:45.902611 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:45.902580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:46.120718 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:46.120689 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:46:51.133497 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.133465 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:46:51.276655 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276550 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7np7z\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-kube-api-access-7np7z\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.276655 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276592 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b75f6a0-8763-4e84-b7bd-83a12633c913-istiod-ca-cert\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.276655 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-envoy\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.276655 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276660 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-token\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.277018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-certs\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.277018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276715 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-credential-socket\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.277018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276752 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-socket\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.277018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276800 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-podinfo\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.277018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.276845 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-data\") pod \"2b75f6a0-8763-4e84-b7bd-83a12633c913\" (UID: \"2b75f6a0-8763-4e84-b7bd-83a12633c913\") " Jul 28 17:46:51.277336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.277049 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:46:51.277336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.277079 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:46:51.277336 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.277224 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:46:51.277504 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.277460 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b75f6a0-8763-4e84-b7bd-83a12633c913-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:46:51.278077 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.278048 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-data" (OuterVolumeSpecName: "istio-data") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:46:51.278960 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.278930 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-token" (OuterVolumeSpecName: "istio-token") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:46:51.278960 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.278951 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:46:51.279083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.278971 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-kube-api-access-7np7z" (OuterVolumeSpecName: "kube-api-access-7np7z") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "kube-api-access-7np7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:46:51.279269 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.279249 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2b75f6a0-8763-4e84-b7bd-83a12633c913" (UID: "2b75f6a0-8763-4e84-b7bd-83a12633c913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:46:51.378241 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378199 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378241 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378232 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378241 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378245 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378256 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378268 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378279 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378290 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7np7z\" (UniqueName: \"kubernetes.io/projected/2b75f6a0-8763-4e84-b7bd-83a12633c913-kube-api-access-7np7z\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378302 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b75f6a0-8763-4e84-b7bd-83a12633c913-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.378506 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.378312 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b75f6a0-8763-4e84-b7bd-83a12633c913-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:46:51.735467 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.735435 2579 generic.go:356] "Generic (PLEG): container finished" podID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerID="bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7" exitCode=0 Jul 28 17:46:51.735671 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.735485 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" Jul 28 17:46:51.735671 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.735518 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" event={"ID":"2b75f6a0-8763-4e84-b7bd-83a12633c913","Type":"ContainerDied","Data":"bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7"} Jul 28 17:46:51.735671 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.735578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd" event={"ID":"2b75f6a0-8763-4e84-b7bd-83a12633c913","Type":"ContainerDied","Data":"47d8e0c4469458dbc60d13af72b51ed05376ee48b47037219b1c6b25fe27eeef"} Jul 28 17:46:51.735671 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.735602 2579 scope.go:117] "RemoveContainer" containerID="bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7" Jul 28 17:46:51.745046 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.745024 2579 scope.go:117] "RemoveContainer" containerID="18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4" Jul 28 17:46:51.753561 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.753540 2579 scope.go:117] "RemoveContainer" containerID="bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7" Jul 28 17:46:51.753896 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:46:51.753875 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7\": container with ID starting with bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7 not found: ID does not exist" containerID="bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7" Jul 28 17:46:51.753940 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.753907 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7"} err="failed to get container status \"bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7\": rpc error: code = NotFound desc = could not find container \"bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7\": container with ID starting with bb4773f02c276fc42a78d02a5ac51c7b4a869e4d480cd7a97bb702c7fa4b95e7 not found: ID does not exist" Jul 28 17:46:51.753940 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.753926 2579 scope.go:117] "RemoveContainer" containerID="18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4" Jul 28 17:46:51.754228 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:46:51.754203 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4\": container with ID starting with 18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4 not found: ID does not exist" containerID="18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4" Jul 28 17:46:51.754318 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.754237 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4"} err="failed to get container status \"18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4\": rpc error: code = NotFound desc = could not find container \"18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4\": container with ID starting with 18632a2bf0f7b617af05064db85e125f654eeedffc18b798a81c83ad4c2d9ef4 not found: ID does not exist" Jul 28 17:46:51.760094 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.760063 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd"] Jul 28 17:46:51.765764 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:51.765740 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7869f9-openshift-default-d6584df46-mmgfd"] Jul 28 17:46:53.684816 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:46:53.684778 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" path="/var/lib/kubelet/pods/2b75f6a0-8763-4e84-b7bd-83a12633c913/volumes" Jul 28 17:47:01.900046 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:01.900008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:02.001711 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:02.001672 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:05.910211 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:05.910170 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:47:06.379248 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:06.379208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:06.844037 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:06.843990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:14.027565 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:14.027525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:14.289123 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:14.288996 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925"] Jul 28 17:47:14.289363 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:14.289333 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" containerID="cri-o://61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509" gracePeriod=30 Jul 28 17:47:14.309872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:14.309843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:14.410677 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:14.410644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:14.574968 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:14.574877 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:14.965610 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:14.965565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:15.259098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.259004 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62"] Jul 28 17:47:15.259518 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.259436 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" Jul 28 17:47:15.259518 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.259452 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" Jul 28 17:47:15.259600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.259544 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e340e962-5b54-4184-b49e-ecaa0bbb3189" containerName="istio-proxy" Jul 28 17:47:15.259600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.259558 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" Jul 28 17:47:15.259600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.259568 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" Jul 28 17:47:15.261958 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.261937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.265981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.265959 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-f0230c-openshift-default-dockercfg-97f4p\"" Jul 28 17:47:15.283049 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.283014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62"] Jul 28 17:47:15.286509 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-certs\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-token\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-credential-socket\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-podinfo\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286660 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286648 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-socket\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-envoy\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf4bq\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-kube-api-access-wf4bq\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-data\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.286821 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.286757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istiod-ca-cert\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.295383 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.295353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:15.387827 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.387786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-certs\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388016 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.387842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-token\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388016 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.387865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-credential-socket\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388016 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.387886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-podinfo\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388016 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.387921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-socket\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388016 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.387942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-envoy\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388016 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.387991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf4bq\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-kube-api-access-wf4bq\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388368 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.388031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-data\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388368 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.388058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istiod-ca-cert\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388368 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.388291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-certs\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388368 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.388348 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-socket\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388602 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.388487 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-credential-socket\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388602 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.388500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-data\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.388704 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.388687 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istiod-ca-cert\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.390308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.390285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-envoy\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.390503 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.390483 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-podinfo\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.393391 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.393362 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:15.408249 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.408220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-token\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.408416 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.408359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf4bq\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-kube-api-access-wf4bq\") pod \"e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.572155 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.571987 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:15.721204 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.721179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62"] Jul 28 17:47:15.723289 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:47:15.723257 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7dd300_afa6_4e3d_b97a_457165cc1fc2.slice/crio-e7a5abbe903b4a2363a123adfbf590b611e3b01796dc18c470fb9129213f6f42 WatchSource:0}: Error finding container e7a5abbe903b4a2363a123adfbf590b611e3b01796dc18c470fb9129213f6f42: Status 404 returned error can't find the container with id e7a5abbe903b4a2363a123adfbf590b611e3b01796dc18c470fb9129213f6f42 Jul 28 17:47:15.725364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.725334 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:47:15.725440 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.725403 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:47:15.725480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.725435 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:47:15.873898 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.873859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" event={"ID":"cc7dd300-afa6-4e3d-b97a-457165cc1fc2","Type":"ContainerStarted","Data":"2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6"} Jul 28 17:47:15.874087 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.873923 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" event={"ID":"cc7dd300-afa6-4e3d-b97a-457165cc1fc2","Type":"ContainerStarted","Data":"e7a5abbe903b4a2363a123adfbf590b611e3b01796dc18c470fb9129213f6f42"} Jul 28 17:47:15.899604 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:15.899545 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" podStartSLOduration=0.899526308 podStartE2EDuration="899.526308ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:15.898157519 +0000 UTC m=+2332.701846110" watchObservedRunningTime="2026-07-28 17:47:15.899526308 +0000 UTC m=+2332.703214875" Jul 28 17:47:16.572258 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:16.572216 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:16.577329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:16.577302 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:16.879854 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:16.879822 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:16.880754 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:16.880732 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:17.955805 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:17.955766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:18.081069 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:18.081037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:18.197335 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:18.197300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:18.307520 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:18.307428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:19.000039 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.000008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:19.117952 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.117917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:19.541071 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.541037 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:47:19.626030 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.625985 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-certs\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626030 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626037 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-envoy\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626076 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-socket\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626170 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-podinfo\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626198 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-credential-socket\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626233 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9465f-7520-46a4-a81e-4a245b9ad423-istiod-ca-cert\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626275 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-token\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626295 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-data\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626339 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4wjc\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-kube-api-access-d4wjc\") pod \"49a9465f-7520-46a4-a81e-4a245b9ad423\" (UID: \"49a9465f-7520-46a4-a81e-4a245b9ad423\") " Jul 28 17:47:19.626646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626380 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:47:19.626646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626521 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:47:19.626646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626549 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.626646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626518 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:47:19.626855 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.626705 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49a9465f-7520-46a4-a81e-4a245b9ad423-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:47:19.627451 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.627421 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-data" (OuterVolumeSpecName: "istio-data") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:47:19.628537 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.628504 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:47:19.628644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.628531 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:47:19.628644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.628538 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-token" (OuterVolumeSpecName: "istio-token") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:47:19.628644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.628597 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-kube-api-access-d4wjc" (OuterVolumeSpecName: "kube-api-access-d4wjc") pod "49a9465f-7520-46a4-a81e-4a245b9ad423" (UID: "49a9465f-7520-46a4-a81e-4a245b9ad423"). InnerVolumeSpecName "kube-api-access-d4wjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:47:19.727610 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727578 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wjc\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-kube-api-access-d4wjc\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.727610 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727607 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.727610 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727618 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.727847 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727626 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.727847 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727635 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.727847 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727643 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49a9465f-7520-46a4-a81e-4a245b9ad423-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.727847 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727651 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.727847 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.727659 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49a9465f-7520-46a4-a81e-4a245b9ad423-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:47:19.906174 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.906054 2579 generic.go:356] "Generic (PLEG): container finished" podID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerID="61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509" exitCode=0 Jul 28 17:47:19.906174 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.906132 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" Jul 28 17:47:19.906174 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.906144 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" event={"ID":"49a9465f-7520-46a4-a81e-4a245b9ad423","Type":"ContainerDied","Data":"61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509"} Jul 28 17:47:19.906417 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.906204 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925" event={"ID":"49a9465f-7520-46a4-a81e-4a245b9ad423","Type":"ContainerDied","Data":"9adfb97ebe7dde9cb8def0b15e63c15a0b6ee2aa37e8f5a28553f2a85dc49067"} Jul 28 17:47:19.906417 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.906226 2579 scope.go:117] "RemoveContainer" containerID="61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509" Jul 28 17:47:19.915447 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.915425 2579 scope.go:117] "RemoveContainer" containerID="f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179" Jul 28 17:47:19.924098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.924075 2579 scope.go:117] "RemoveContainer" containerID="61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509" Jul 28 17:47:19.924442 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:47:19.924421 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509\": container with ID starting with 61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509 not found: ID does not exist" containerID="61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509" Jul 28 17:47:19.924494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.924454 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509"} err="failed to get container status \"61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509\": rpc error: code = NotFound desc = could not find container \"61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509\": container with ID starting with 61f1b7df5e98d9760206142cb07c65dbebddc1819b995f2b3774760b8262b509 not found: ID does not exist" Jul 28 17:47:19.924494 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.924475 2579 scope.go:117] "RemoveContainer" containerID="f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179" Jul 28 17:47:19.924752 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:47:19.924733 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179\": container with ID starting with f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179 not found: ID does not exist" containerID="f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179" Jul 28 17:47:19.924789 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.924769 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179"} err="failed to get container status \"f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179\": rpc error: code = NotFound desc = could not find container \"f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179\": container with ID starting with f1a13071b8d14350b0b1765ae8015bb7495337a841ac35ddfdc7c43ef3341179 not found: ID does not exist" Jul 28 17:47:19.935303 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.935266 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925"] Jul 28 17:47:19.946678 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:19.946645 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-29d9e9-openshift-default-67f445fbf8-gb925"] Jul 28 17:47:21.689763 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:21.689732 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" path="/var/lib/kubelet/pods/49a9465f-7520-46a4-a81e-4a245b9ad423/volumes" Jul 28 17:47:22.403643 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:22.403611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:22.784159 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:22.784042 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:23.203975 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:23.203940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:23.632350 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:23.632312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:25.325224 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:25.325183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:47:26.595000 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:26.594966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:26.870186 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:26.870085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:27.672250 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:27.672211 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:27.983613 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:27.983526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:28.205116 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:28.205086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:31.051673 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:31.051585 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:32.921345 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:32.921296 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 17:47:32.921801 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:32.921379 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 17:47:33.981315 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:33.981279 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerID="2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6" exitCode=0 Jul 28 17:47:33.981874 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:33.981357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" event={"ID":"cc7dd300-afa6-4e3d-b97a-457165cc1fc2","Type":"ContainerDied","Data":"2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6"} Jul 28 17:47:33.981874 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:33.981843 2579 scope.go:117] "RemoveContainer" containerID="2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6" Jul 28 17:47:33.982833 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:33.982808 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:47:33.982898 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:33.982873 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:47:33.982938 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:33.982903 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:47:34.987566 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:34.987532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" event={"ID":"cc7dd300-afa6-4e3d-b97a-457165cc1fc2","Type":"ContainerStarted","Data":"c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423"} Jul 28 17:47:35.572737 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:35.572690 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:35.572993 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:35.572969 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 28 17:47:35.573056 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:35.573034 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 28 17:47:36.572850 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:36.572806 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 28 17:47:36.573405 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:36.572876 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 28 17:47:37.572773 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:37.572735 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 28 17:47:37.573051 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:37.572805 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 28 17:47:38.229296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:38.229256 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:47:38.573017 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:38.572915 2579 patch_prober.go:28] interesting pod/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 28 17:47:38.573017 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:38.572975 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 28 17:47:39.576521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:39.576487 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:39.576934 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:39.576824 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:39.577632 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:39.577613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 17:47:41.547445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:41.547396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:47:48.734125 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:47:48.734067 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:48:02.005271 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005229 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp"] Jul 28 17:48:02.005754 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005722 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" Jul 28 17:48:02.005754 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005740 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" Jul 28 17:48:02.005754 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005751 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" Jul 28 17:48:02.005918 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005760 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b75f6a0-8763-4e84-b7bd-83a12633c913" containerName="istio-proxy" Jul 28 17:48:02.005918 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005780 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" Jul 28 17:48:02.005918 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005789 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" Jul 28 17:48:02.005918 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.005875 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" Jul 28 17:48:02.007769 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.007747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.010994 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.010968 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"auth-test-model-f24b7c-kserve-self-signed-certs\"" Jul 28 17:48:02.011117 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.010973 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"openshift-service-ca.crt\"" Jul 28 17:48:02.012884 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.012639 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"kube-root-ca.crt\"" Jul 28 17:48:02.012884 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.012740 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f24b7c\"/\"default-dockercfg-77rkw\"" Jul 28 17:48:02.022389 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.022360 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp"] Jul 28 17:48:02.042999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.042937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:48:02.106329 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.106288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-model-cache\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.106544 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.106341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-tmp-dir\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.106544 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.106434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbw5s\" (UniqueName: \"kubernetes.io/projected/e67857ea-ee84-45d0-bb16-4072027438ae-kube-api-access-bbw5s\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.106544 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.106474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-home\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.106544 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.106501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67857ea-ee84-45d0-bb16-4072027438ae-tls-certs\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.106700 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.106568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-dshm\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.207496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.207450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-dshm\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.207496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.207502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-model-cache\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.207771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.207533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-tmp-dir\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.207771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.207563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbw5s\" (UniqueName: \"kubernetes.io/projected/e67857ea-ee84-45d0-bb16-4072027438ae-kube-api-access-bbw5s\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.207771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.207585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-home\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.207771 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.207600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67857ea-ee84-45d0-bb16-4072027438ae-tls-certs\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.207997 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.207972 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-model-cache\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.208060 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.208036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-home\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.208098 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.208046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-tmp-dir\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.209884 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.209859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-dshm\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.210033 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.210016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67857ea-ee84-45d0-bb16-4072027438ae-tls-certs\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.218048 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.218024 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbw5s\" (UniqueName: \"kubernetes.io/projected/e67857ea-ee84-45d0-bb16-4072027438ae-kube-api-access-bbw5s\") pod \"auth-test-model-f24b7c-kserve-b87964bff-mzxbp\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.320944 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.320846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:02.458028 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:02.457999 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp"] Jul 28 17:48:02.459787 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:48:02.459760 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode67857ea_ee84_45d0_bb16_4072027438ae.slice/crio-2b2f0896b68289a87daca1f7d557521e30969fa1492dc8a833912576858f5a0e WatchSource:0}: Error finding container 2b2f0896b68289a87daca1f7d557521e30969fa1492dc8a833912576858f5a0e: Status 404 returned error can't find the container with id 2b2f0896b68289a87daca1f7d557521e30969fa1492dc8a833912576858f5a0e Jul 28 17:48:03.129173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:03.129130 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" event={"ID":"e67857ea-ee84-45d0-bb16-4072027438ae","Type":"ContainerStarted","Data":"6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b"} Jul 28 17:48:03.129636 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:03.129189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" event={"ID":"e67857ea-ee84-45d0-bb16-4072027438ae","Type":"ContainerStarted","Data":"2b2f0896b68289a87daca1f7d557521e30969fa1492dc8a833912576858f5a0e"} Jul 28 17:48:03.152202 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:03.152147 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" podStartSLOduration=2.152088388 podStartE2EDuration="2.152088388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:03.15122892 +0000 UTC m=+2379.954917527" watchObservedRunningTime="2026-07-28 17:48:03.152088388 +0000 UTC m=+2379.955776978" Jul 28 17:48:06.319518 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:06.319477 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:48:12.322063 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:12.322023 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:12.322597 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:12.322082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:12.334485 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:12.334454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:13.188868 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:13.188836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:48:23.922814 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:23.922778 2579 scope.go:117] "RemoveContainer" containerID="b611bb809381f687479b0ae02ca292c27a2f5e14e6051740c137de3b44ca93e7" Jul 28 17:48:24.171319 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:24.171288 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:48:24.182652 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:24.182582 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:48:43.003409 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:48:43.003365 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:49:05.107608 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:49:05.107570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:49:23.001484 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:49:23.001445 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:49:46.724951 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:49:46.724915 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:50:02.639445 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:50:02.639407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:50:31.000884 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:50:31.000781 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:50:49.212554 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:50:49.212510 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:51:16.503744 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:51:16.503702 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:51:31.607073 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:51:31.607028 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:51:39.395525 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:51:39.395490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:51:42.908626 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:51:42.908592 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:52:00.095809 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:00.095720 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:52:04.605470 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:04.605430 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:52:06.216516 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:06.216478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:52:21.718625 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:21.718585 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:52:23.968031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:23.967996 2579 scope.go:117] "RemoveContainer" containerID="4082981fd7ce7f372ff398901ea28b6060af4e8f18893faf24e60e7ba2adec2d" Jul 28 17:52:47.520872 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:47.520836 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:52:50.522617 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:50.522576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:52:51.635061 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:51.635021 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp"] Jul 28 17:52:51.655420 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:52:51.655384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:53:00.006207 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.006164 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h"] Jul 28 17:53:00.006694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.006675 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="49a9465f-7520-46a4-a81e-4a245b9ad423" containerName="istio-proxy" Jul 28 17:53:00.009337 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.009320 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.012433 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.012404 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"kube-root-ca.crt\"" Jul 28 17:53:00.013561 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.013538 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"default-dockercfg-gjbh8\"" Jul 28 17:53:00.013688 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.013570 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"openshift-service-ca.crt\"" Jul 28 17:53:00.013688 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.013624 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"sub-test-model-f0230c-kserve-self-signed-certs\"" Jul 28 17:53:00.020283 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.020255 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h"] Jul 28 17:53:00.042887 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.042854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:53:00.136607 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.136563 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-model-cache\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.136818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.136613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98895eb-f149-4d2b-a239-00174b902497-tls-certs\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.136818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.136694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-tmp-dir\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.136818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.136743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-home\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.136818 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.136786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-dshm\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.136998 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.136836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h2s4\" (UniqueName: \"kubernetes.io/projected/d98895eb-f149-4d2b-a239-00174b902497-kube-api-access-6h2s4\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238076 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-model-cache\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238076 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98895eb-f149-4d2b-a239-00174b902497-tls-certs\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238076 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-tmp-dir\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238523 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-home\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238523 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-dshm\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238523 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h2s4\" (UniqueName: \"kubernetes.io/projected/d98895eb-f149-4d2b-a239-00174b902497-kube-api-access-6h2s4\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238664 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238564 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-model-cache\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238664 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-home\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.238664 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.238647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-tmp-dir\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.240592 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.240572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-dshm\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.240880 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.240860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98895eb-f149-4d2b-a239-00174b902497-tls-certs\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.247819 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.247788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h2s4\" (UniqueName: \"kubernetes.io/projected/d98895eb-f149-4d2b-a239-00174b902497-kube-api-access-6h2s4\") pod \"sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.321585 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.321491 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:00.465710 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.465678 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h"] Jul 28 17:53:00.467517 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:53:00.467488 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd98895eb_f149_4d2b_a239_00174b902497.slice/crio-b75a26d0533568d199e561a8658d0fdbd533e083cb10ba470cf0748cbc738eb8 WatchSource:0}: Error finding container b75a26d0533568d199e561a8658d0fdbd533e083cb10ba470cf0748cbc738eb8: Status 404 returned error can't find the container with id b75a26d0533568d199e561a8658d0fdbd533e083cb10ba470cf0748cbc738eb8 Jul 28 17:53:00.603644 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.603603 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" event={"ID":"d98895eb-f149-4d2b-a239-00174b902497","Type":"ContainerStarted","Data":"2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19"} Jul 28 17:53:00.603824 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.603651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" event={"ID":"d98895eb-f149-4d2b-a239-00174b902497","Type":"ContainerStarted","Data":"b75a26d0533568d199e561a8658d0fdbd533e083cb10ba470cf0748cbc738eb8"} Jul 28 17:53:00.631540 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:00.631479 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" podStartSLOduration=1.6314581989999999 podStartE2EDuration="1.631458199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:00.629857014 +0000 UTC m=+2677.433545618" watchObservedRunningTime="2026-07-28 17:53:00.631458199 +0000 UTC m=+2677.435146791" Jul 28 17:53:03.910895 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:03.910852 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:53:06.653740 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:06.653668 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" podUID="e67857ea-ee84-45d0-bb16-4072027438ae" containerName="main" containerID="cri-o://6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b" gracePeriod=45 Jul 28 17:53:06.913939 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:06.913857 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:53:07.004235 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004197 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-home\") pod \"e67857ea-ee84-45d0-bb16-4072027438ae\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " Jul 28 17:53:07.004235 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004239 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbw5s\" (UniqueName: \"kubernetes.io/projected/e67857ea-ee84-45d0-bb16-4072027438ae-kube-api-access-bbw5s\") pod \"e67857ea-ee84-45d0-bb16-4072027438ae\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " Jul 28 17:53:07.004490 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-tmp-dir\") pod \"e67857ea-ee84-45d0-bb16-4072027438ae\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " Jul 28 17:53:07.004490 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004326 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-model-cache\") pod \"e67857ea-ee84-45d0-bb16-4072027438ae\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " Jul 28 17:53:07.004490 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004344 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67857ea-ee84-45d0-bb16-4072027438ae-tls-certs\") pod \"e67857ea-ee84-45d0-bb16-4072027438ae\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " Jul 28 17:53:07.004490 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004382 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-dshm\") pod \"e67857ea-ee84-45d0-bb16-4072027438ae\" (UID: \"e67857ea-ee84-45d0-bb16-4072027438ae\") " Jul 28 17:53:07.004690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004543 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-home" (OuterVolumeSpecName: "home") pod "e67857ea-ee84-45d0-bb16-4072027438ae" (UID: "e67857ea-ee84-45d0-bb16-4072027438ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:07.004690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004564 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e67857ea-ee84-45d0-bb16-4072027438ae" (UID: "e67857ea-ee84-45d0-bb16-4072027438ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:07.004690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004604 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "e67857ea-ee84-45d0-bb16-4072027438ae" (UID: "e67857ea-ee84-45d0-bb16-4072027438ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:07.004690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004666 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-home\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:53:07.004690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004677 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-tmp-dir\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:53:07.004690 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.004687 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-model-cache\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:53:07.006694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.006656 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-dshm" (OuterVolumeSpecName: "dshm") pod "e67857ea-ee84-45d0-bb16-4072027438ae" (UID: "e67857ea-ee84-45d0-bb16-4072027438ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:53:07.006694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.006664 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67857ea-ee84-45d0-bb16-4072027438ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e67857ea-ee84-45d0-bb16-4072027438ae" (UID: "e67857ea-ee84-45d0-bb16-4072027438ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:53:07.006861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.006703 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67857ea-ee84-45d0-bb16-4072027438ae-kube-api-access-bbw5s" (OuterVolumeSpecName: "kube-api-access-bbw5s") pod "e67857ea-ee84-45d0-bb16-4072027438ae" (UID: "e67857ea-ee84-45d0-bb16-4072027438ae"). InnerVolumeSpecName "kube-api-access-bbw5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:53:07.105513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.105450 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67857ea-ee84-45d0-bb16-4072027438ae-tls-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:53:07.105513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.105499 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67857ea-ee84-45d0-bb16-4072027438ae-dshm\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:53:07.105513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.105511 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbw5s\" (UniqueName: \"kubernetes.io/projected/e67857ea-ee84-45d0-bb16-4072027438ae-kube-api-access-bbw5s\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:53:07.639400 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.639359 2579 generic.go:356] "Generic (PLEG): container finished" podID="e67857ea-ee84-45d0-bb16-4072027438ae" containerID="6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b" exitCode=0 Jul 28 17:53:07.639595 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.639407 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" Jul 28 17:53:07.639595 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.639412 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" event={"ID":"e67857ea-ee84-45d0-bb16-4072027438ae","Type":"ContainerDied","Data":"6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b"} Jul 28 17:53:07.639595 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.639526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp" event={"ID":"e67857ea-ee84-45d0-bb16-4072027438ae","Type":"ContainerDied","Data":"2b2f0896b68289a87daca1f7d557521e30969fa1492dc8a833912576858f5a0e"} Jul 28 17:53:07.639595 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.639551 2579 scope.go:117] "RemoveContainer" containerID="6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b" Jul 28 17:53:07.649289 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.649269 2579 scope.go:117] "RemoveContainer" containerID="6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b" Jul 28 17:53:07.649604 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:53:07.649582 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b\": container with ID starting with 6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b not found: ID does not exist" containerID="6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b" Jul 28 17:53:07.649646 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.649617 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b"} err="failed to get container status \"6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b\": rpc error: code = NotFound desc = could not find container \"6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b\": container with ID starting with 6cdd0220300206b1ed3dd80dbdca53b31c3af837a361b7625fcf5da41233e96b not found: ID does not exist" Jul 28 17:53:07.665848 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.665810 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp"] Jul 28 17:53:07.674178 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.674141 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f24b7c/auth-test-model-f24b7c-kserve-b87964bff-mzxbp"] Jul 28 17:53:07.685338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:07.685300 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e67857ea-ee84-45d0-bb16-4072027438ae" path="/var/lib/kubelet/pods/e67857ea-ee84-45d0-bb16-4072027438ae/volumes" Jul 28 17:53:10.322287 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:10.322247 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:10.322718 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:10.322309 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:10.334943 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:10.334908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:10.668577 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:10.668542 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:53:22.261152 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:22.260993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:53:24.205323 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:24.205292 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:53:24.214995 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:24.214966 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:53:25.822424 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:25.822384 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:53:55.111019 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:55.110979 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:53:58.501385 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:58.501295 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:53:58.585037 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:53:58.584992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:54:12.840468 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:12.840429 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:54:15.205439 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:15.205397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:54:15.310601 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:15.310566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:54:33.135279 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:33.135239 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:54:39.437778 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:39.437733 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h"] Jul 28 17:54:39.467614 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:39.467572 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:54:40.354550 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:40.354514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:54:45.411994 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.411954 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k"] Jul 28 17:54:45.412508 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.412283 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67857ea-ee84-45d0-bb16-4072027438ae" containerName="main" Jul 28 17:54:45.412508 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.412299 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67857ea-ee84-45d0-bb16-4072027438ae" containerName="main" Jul 28 17:54:45.412508 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.412366 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67857ea-ee84-45d0-bb16-4072027438ae" containerName="main" Jul 28 17:54:45.415338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.415318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.418910 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.418885 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-f0230c\"/\"rate-test-model-f0230c-kserve-self-signed-certs\"" Jul 28 17:54:45.422411 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.422383 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:54:45.427199 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.427165 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k"] Jul 28 17:54:45.445532 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.445496 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-tmp-dir\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.445532 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.445536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-home\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.445815 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.445575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-model-cache\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.445815 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.445608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59668006-2d2b-47fb-86a0-d117209ec8fa-tls-certs\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.445815 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.445643 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-dshm\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.445815 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.445665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n958t\" (UniqueName: \"kubernetes.io/projected/59668006-2d2b-47fb-86a0-d117209ec8fa-kube-api-access-n958t\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547039 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.546953 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-model-cache\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547039 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.546990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59668006-2d2b-47fb-86a0-d117209ec8fa-tls-certs\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547039 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.547012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-dshm\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.547147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n958t\" (UniqueName: \"kubernetes.io/projected/59668006-2d2b-47fb-86a0-d117209ec8fa-kube-api-access-n958t\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.547265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-tmp-dir\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.547297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-home\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547513 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.547493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-model-cache\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547584 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.547569 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-home\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.547661 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.547631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-tmp-dir\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.549338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.549318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-dshm\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.549488 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.549470 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59668006-2d2b-47fb-86a0-d117209ec8fa-tls-certs\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.563291 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.563253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n958t\" (UniqueName: \"kubernetes.io/projected/59668006-2d2b-47fb-86a0-d117209ec8fa-kube-api-access-n958t\") pod \"rate-test-model-f0230c-kserve-84fdc685f-mdg6k\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.725026 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.724919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:45.868928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:45.868897 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k"] Jul 28 17:54:45.871611 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:54:45.871587 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59668006_2d2b_47fb_86a0_d117209ec8fa.slice/crio-d1756752aa6e48fbffe57ac28410c5841795827fee1df0fd1e9e8dc8d35cc74c WatchSource:0}: Error finding container d1756752aa6e48fbffe57ac28410c5841795827fee1df0fd1e9e8dc8d35cc74c: Status 404 returned error can't find the container with id d1756752aa6e48fbffe57ac28410c5841795827fee1df0fd1e9e8dc8d35cc74c Jul 28 17:54:46.128165 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:46.128090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" event={"ID":"59668006-2d2b-47fb-86a0-d117209ec8fa","Type":"ContainerStarted","Data":"acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053"} Jul 28 17:54:46.128358 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:46.128177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" event={"ID":"59668006-2d2b-47fb-86a0-d117209ec8fa","Type":"ContainerStarted","Data":"d1756752aa6e48fbffe57ac28410c5841795827fee1df0fd1e9e8dc8d35cc74c"} Jul 28 17:54:46.152861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:46.152807 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" podStartSLOduration=1.152788625 podStartE2EDuration="1.152788625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:46.151549058 +0000 UTC m=+2782.955237648" watchObservedRunningTime="2026-07-28 17:54:46.152788625 +0000 UTC m=+2782.956477215" Jul 28 17:54:51.498296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:51.498256 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:54:54.456774 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.456726 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" podUID="d98895eb-f149-4d2b-a239-00174b902497" containerName="main" containerID="cri-o://2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19" gracePeriod=45 Jul 28 17:54:54.710849 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.710773 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:54:54.835334 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835295 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6h2s4\" (UniqueName: \"kubernetes.io/projected/d98895eb-f149-4d2b-a239-00174b902497-kube-api-access-6h2s4\") pod \"d98895eb-f149-4d2b-a239-00174b902497\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " Jul 28 17:54:54.835334 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835339 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-dshm\") pod \"d98895eb-f149-4d2b-a239-00174b902497\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " Jul 28 17:54:54.835569 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835385 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98895eb-f149-4d2b-a239-00174b902497-tls-certs\") pod \"d98895eb-f149-4d2b-a239-00174b902497\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " Jul 28 17:54:54.835569 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835415 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-tmp-dir\") pod \"d98895eb-f149-4d2b-a239-00174b902497\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " Jul 28 17:54:54.835569 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835447 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-model-cache\") pod \"d98895eb-f149-4d2b-a239-00174b902497\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " Jul 28 17:54:54.835715 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-home\") pod \"d98895eb-f149-4d2b-a239-00174b902497\" (UID: \"d98895eb-f149-4d2b-a239-00174b902497\") " Jul 28 17:54:54.835772 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835716 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d98895eb-f149-4d2b-a239-00174b902497" (UID: "d98895eb-f149-4d2b-a239-00174b902497"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:54.835772 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835728 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-model-cache" (OuterVolumeSpecName: "model-cache") pod "d98895eb-f149-4d2b-a239-00174b902497" (UID: "d98895eb-f149-4d2b-a239-00174b902497"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:54.835946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.835922 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-home" (OuterVolumeSpecName: "home") pod "d98895eb-f149-4d2b-a239-00174b902497" (UID: "d98895eb-f149-4d2b-a239-00174b902497"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:54.836165 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.836145 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-home\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:54:54.836271 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.836168 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-tmp-dir\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:54:54.836271 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.836182 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-model-cache\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:54:54.837815 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.837789 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-dshm" (OuterVolumeSpecName: "dshm") pod "d98895eb-f149-4d2b-a239-00174b902497" (UID: "d98895eb-f149-4d2b-a239-00174b902497"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:54:54.837908 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.837811 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d98895eb-f149-4d2b-a239-00174b902497-kube-api-access-6h2s4" (OuterVolumeSpecName: "kube-api-access-6h2s4") pod "d98895eb-f149-4d2b-a239-00174b902497" (UID: "d98895eb-f149-4d2b-a239-00174b902497"). InnerVolumeSpecName "kube-api-access-6h2s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:54:54.837908 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.837825 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d98895eb-f149-4d2b-a239-00174b902497-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d98895eb-f149-4d2b-a239-00174b902497" (UID: "d98895eb-f149-4d2b-a239-00174b902497"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:54:54.937489 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.937447 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d98895eb-f149-4d2b-a239-00174b902497-tls-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:54:54.937489 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.937486 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h2s4\" (UniqueName: \"kubernetes.io/projected/d98895eb-f149-4d2b-a239-00174b902497-kube-api-access-6h2s4\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:54:54.937711 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:54.937499 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d98895eb-f149-4d2b-a239-00174b902497-dshm\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:54:55.177437 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.177388 2579 generic.go:356] "Generic (PLEG): container finished" podID="d98895eb-f149-4d2b-a239-00174b902497" containerID="2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19" exitCode=0 Jul 28 17:54:55.177612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.177470 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" Jul 28 17:54:55.177612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.177481 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" event={"ID":"d98895eb-f149-4d2b-a239-00174b902497","Type":"ContainerDied","Data":"2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19"} Jul 28 17:54:55.177612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.177540 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h" event={"ID":"d98895eb-f149-4d2b-a239-00174b902497","Type":"ContainerDied","Data":"b75a26d0533568d199e561a8658d0fdbd533e083cb10ba470cf0748cbc738eb8"} Jul 28 17:54:55.177612 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.177559 2579 scope.go:117] "RemoveContainer" containerID="2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19" Jul 28 17:54:55.187492 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.187469 2579 scope.go:117] "RemoveContainer" containerID="2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19" Jul 28 17:54:55.187813 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:54:55.187791 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19\": container with ID starting with 2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19 not found: ID does not exist" containerID="2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19" Jul 28 17:54:55.187878 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.187823 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19"} err="failed to get container status \"2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19\": rpc error: code = NotFound desc = could not find container \"2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19\": container with ID starting with 2752bfff71e71f6c8b39486b295dd882999d5d07097210e6bd33265f15f3ab19 not found: ID does not exist" Jul 28 17:54:55.203633 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.203600 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h"] Jul 28 17:54:55.209743 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.209711 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/sub-test-model-f0230c-kserve-6c65cdddbc-m7m2h"] Jul 28 17:54:55.685623 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.685584 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d98895eb-f149-4d2b-a239-00174b902497" path="/var/lib/kubelet/pods/d98895eb-f149-4d2b-a239-00174b902497/volumes" Jul 28 17:54:55.726044 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.726005 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:55.726252 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.726059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:55.738808 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:55.738779 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:54:56.196728 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:54:56.196697 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:55:01.313293 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:55:01.313244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:55:01.415198 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:55:01.415168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:55:08.339308 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:55:08.339272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:55:12.104775 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:55:12.104730 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:55:54.706556 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:55:54.706520 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:56:12.496201 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:12.496158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:56:32.501999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:32.501904 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:56:39.489290 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:39.489251 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k"] Jul 28 17:56:39.557255 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:39.557215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:56:40.418425 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:40.418385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:56:45.815482 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:45.815447 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:56:54.508230 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.508179 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" podUID="59668006-2d2b-47fb-86a0-d117209ec8fa" containerName="main" containerID="cri-o://acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053" gracePeriod=45 Jul 28 17:56:54.756627 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.756593 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:56:54.762169 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.762077 2579 generic.go:356] "Generic (PLEG): container finished" podID="59668006-2d2b-47fb-86a0-d117209ec8fa" containerID="acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053" exitCode=0 Jul 28 17:56:54.762169 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.762133 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" Jul 28 17:56:54.762169 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.762140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" event={"ID":"59668006-2d2b-47fb-86a0-d117209ec8fa","Type":"ContainerDied","Data":"acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053"} Jul 28 17:56:54.762431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.762189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k" event={"ID":"59668006-2d2b-47fb-86a0-d117209ec8fa","Type":"ContainerDied","Data":"d1756752aa6e48fbffe57ac28410c5841795827fee1df0fd1e9e8dc8d35cc74c"} Jul 28 17:56:54.762431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.762216 2579 scope.go:117] "RemoveContainer" containerID="acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053" Jul 28 17:56:54.770607 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.770584 2579 scope.go:117] "RemoveContainer" containerID="acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053" Jul 28 17:56:54.770927 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:56:54.770905 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053\": container with ID starting with acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053 not found: ID does not exist" containerID="acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053" Jul 28 17:56:54.770981 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.770938 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053"} err="failed to get container status \"acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053\": rpc error: code = NotFound desc = could not find container \"acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053\": container with ID starting with acd356ba6a07974616fae6b57353ff899a282ad51cf9ad3a7f11c9926f5c7053 not found: ID does not exist" Jul 28 17:56:54.889668 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.889621 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-dshm\") pod \"59668006-2d2b-47fb-86a0-d117209ec8fa\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " Jul 28 17:56:54.889855 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.889702 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-tmp-dir\") pod \"59668006-2d2b-47fb-86a0-d117209ec8fa\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " Jul 28 17:56:54.889855 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.889766 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-home\") pod \"59668006-2d2b-47fb-86a0-d117209ec8fa\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " Jul 28 17:56:54.889928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.889872 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59668006-2d2b-47fb-86a0-d117209ec8fa-tls-certs\") pod \"59668006-2d2b-47fb-86a0-d117209ec8fa\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " Jul 28 17:56:54.889928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.889907 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n958t\" (UniqueName: \"kubernetes.io/projected/59668006-2d2b-47fb-86a0-d117209ec8fa-kube-api-access-n958t\") pod \"59668006-2d2b-47fb-86a0-d117209ec8fa\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " Jul 28 17:56:54.890023 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.889953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-model-cache\") pod \"59668006-2d2b-47fb-86a0-d117209ec8fa\" (UID: \"59668006-2d2b-47fb-86a0-d117209ec8fa\") " Jul 28 17:56:54.890023 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.889973 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-home" (OuterVolumeSpecName: "home") pod "59668006-2d2b-47fb-86a0-d117209ec8fa" (UID: "59668006-2d2b-47fb-86a0-d117209ec8fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:56:54.890158 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.890029 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59668006-2d2b-47fb-86a0-d117209ec8fa" (UID: "59668006-2d2b-47fb-86a0-d117209ec8fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:56:54.890296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.890272 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "59668006-2d2b-47fb-86a0-d117209ec8fa" (UID: "59668006-2d2b-47fb-86a0-d117209ec8fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:56:54.890296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.890279 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-tmp-dir\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:56:54.890422 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.890309 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-home\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:56:54.891943 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.891919 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-dshm" (OuterVolumeSpecName: "dshm") pod "59668006-2d2b-47fb-86a0-d117209ec8fa" (UID: "59668006-2d2b-47fb-86a0-d117209ec8fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:56:54.892038 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.891951 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59668006-2d2b-47fb-86a0-d117209ec8fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59668006-2d2b-47fb-86a0-d117209ec8fa" (UID: "59668006-2d2b-47fb-86a0-d117209ec8fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:56:54.892038 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.891962 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59668006-2d2b-47fb-86a0-d117209ec8fa-kube-api-access-n958t" (OuterVolumeSpecName: "kube-api-access-n958t") pod "59668006-2d2b-47fb-86a0-d117209ec8fa" (UID: "59668006-2d2b-47fb-86a0-d117209ec8fa"). InnerVolumeSpecName "kube-api-access-n958t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:56:54.990783 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.990740 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-dshm\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:56:54.990978 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.990795 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59668006-2d2b-47fb-86a0-d117209ec8fa-tls-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:56:54.990978 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.990809 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n958t\" (UniqueName: \"kubernetes.io/projected/59668006-2d2b-47fb-86a0-d117209ec8fa-kube-api-access-n958t\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:56:54.990978 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:54.990821 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59668006-2d2b-47fb-86a0-d117209ec8fa-model-cache\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:56:55.086619 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:55.086579 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k"] Jul 28 17:56:55.091500 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:55.091462 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-f0230c/rate-test-model-f0230c-kserve-84fdc685f-mdg6k"] Jul 28 17:56:55.684182 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:56:55.684150 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59668006-2d2b-47fb-86a0-d117209ec8fa" path="/var/lib/kubelet/pods/59668006-2d2b-47fb-86a0-d117209ec8fa/volumes" Jul 28 17:57:00.478300 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.478268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:00.858274 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.858228 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd"] Jul 28 17:57:00.858567 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.858552 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d98895eb-f149-4d2b-a239-00174b902497" containerName="main" Jul 28 17:57:00.858567 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.858568 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d98895eb-f149-4d2b-a239-00174b902497" containerName="main" Jul 28 17:57:00.858707 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.858575 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59668006-2d2b-47fb-86a0-d117209ec8fa" containerName="main" Jul 28 17:57:00.858707 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.858581 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="59668006-2d2b-47fb-86a0-d117209ec8fa" containerName="main" Jul 28 17:57:00.858707 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.858637 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d98895eb-f149-4d2b-a239-00174b902497" containerName="main" Jul 28 17:57:00.858707 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.858645 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="59668006-2d2b-47fb-86a0-d117209ec8fa" containerName="main" Jul 28 17:57:00.863002 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.862979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.865940 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.865917 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-20ce8c-openshift-default-dockercfg-prmlc\"" Jul 28 17:57:00.877862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.877826 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd"] Jul 28 17:57:00.937776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.937739 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9mjp\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-kube-api-access-k9mjp\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.937776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.937782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-credential-socket\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.937999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.937820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-data\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.937999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.937842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b85d9-1e44-4ab2-bdad-213169862e53-istiod-ca-cert\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.937999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.937863 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-socket\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.937999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.937940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-certs\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.937999 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.937980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-envoy\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.938244 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.938028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b85d9-1e44-4ab2-bdad-213169862e53-istio-podinfo\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:00.938244 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:00.938053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-istio-token\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.038591 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-certs\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.038591 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-envoy\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.038862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b85d9-1e44-4ab2-bdad-213169862e53-istio-podinfo\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.038862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-istio-token\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.038862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9mjp\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-kube-api-access-k9mjp\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.038862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-credential-socket\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.038862 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-data\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.039147 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.038993 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-certs\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.039147 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.039009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b85d9-1e44-4ab2-bdad-213169862e53-istiod-ca-cert\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.039147 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.039059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-socket\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.039262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.039226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-credential-socket\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.039338 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.039316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-data\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.039420 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.039400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-socket\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.039670 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.039655 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b85d9-1e44-4ab2-bdad-213169862e53-istiod-ca-cert\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.041100 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.041070 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-envoy\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.041545 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.041523 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b85d9-1e44-4ab2-bdad-213169862e53-istio-podinfo\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.052150 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.052121 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9mjp\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-kube-api-access-k9mjp\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.054413 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.054390 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-istio-token\") pod \"e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.173181 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.173070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:01.324774 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.324745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd"] Jul 28 17:57:01.326202 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:57:01.326173 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357b85d9_1e44_4ab2_bdad_213169862e53.slice/crio-83f12555baf2c8077ffd9bfcbd25749e6a2f68daf4ea214cf8e2f3295f2038be WatchSource:0}: Error finding container 83f12555baf2c8077ffd9bfcbd25749e6a2f68daf4ea214cf8e2f3295f2038be: Status 404 returned error can't find the container with id 83f12555baf2c8077ffd9bfcbd25749e6a2f68daf4ea214cf8e2f3295f2038be Jul 28 17:57:01.328297 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.328264 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:57:01.328362 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.328337 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:57:01.328400 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.328366 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:57:01.799439 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.799397 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" event={"ID":"357b85d9-1e44-4ab2-bdad-213169862e53","Type":"ContainerStarted","Data":"998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849"} Jul 28 17:57:01.799876 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.799452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" event={"ID":"357b85d9-1e44-4ab2-bdad-213169862e53","Type":"ContainerStarted","Data":"83f12555baf2c8077ffd9bfcbd25749e6a2f68daf4ea214cf8e2f3295f2038be"} Jul 28 17:57:01.828539 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:01.828482 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podStartSLOduration=1.828433988 podStartE2EDuration="1.828433988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:01.827956416 +0000 UTC m=+2918.631645008" watchObservedRunningTime="2026-07-28 17:57:01.828433988 +0000 UTC m=+2918.632146827" Jul 28 17:57:02.174275 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:02.174237 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:02.179275 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:02.179243 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:02.208485 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:02.208451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:02.805431 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:02.805396 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:02.806483 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:02.806462 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:03.808328 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:03.808296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:03.906163 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:03.906124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:04.017081 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:04.017047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:04.454894 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:04.454855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:04.614626 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:04.614591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:04.717694 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:04.717606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:08.564545 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:08.564506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:12.428831 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:12.428789 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:57:18.847291 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:18.847248 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 17:57:18.847725 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:18.847340 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 17:57:19.241987 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:19.241887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:19.603584 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:19.603543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:20.147245 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:20.147213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:20.907119 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:20.907084 2579 generic.go:356] "Generic (PLEG): container finished" podID="357b85d9-1e44-4ab2-bdad-213169862e53" containerID="998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849" exitCode=0 Jul 28 17:57:20.907339 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:20.907159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" event={"ID":"357b85d9-1e44-4ab2-bdad-213169862e53","Type":"ContainerDied","Data":"998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849"} Jul 28 17:57:20.907567 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:20.907553 2579 scope.go:117] "RemoveContainer" containerID="998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849" Jul 28 17:57:20.908611 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:20.908586 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:57:20.908679 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:20.908653 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:57:20.908729 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:20.908680 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:57:21.914446 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:21.914407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" event={"ID":"357b85d9-1e44-4ab2-bdad-213169862e53","Type":"ContainerStarted","Data":"c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86"} Jul 28 17:57:22.174047 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:22.173935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:22.174275 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:22.174254 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 28 17:57:22.174365 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:22.174330 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 28 17:57:23.008675 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.008642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:23.107945 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.107904 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:23.174230 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.174189 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 28 17:57:23.174421 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.174260 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 28 17:57:23.250083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.250041 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf"] Jul 28 17:57:23.255374 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.255338 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.264678 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.264250 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-857ec3-serving-cert\"" Jul 28 17:57:23.265521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.265498 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\"" Jul 28 17:57:23.265521 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.265511 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 17:57:23.265739 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.265503 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 17:57:23.283776 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.283742 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf"] Jul 28 17:57:23.309997 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.309965 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:23.340327 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.340052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-kube-api-access-xdn2q\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.340327 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.340169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.340327 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.340241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.441341 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.441288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.441573 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.441391 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.441573 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.441431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-kube-api-access-xdn2q\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.443026 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.442991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.447248 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.447217 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:23.447957 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.447929 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.463233 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.463199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-kube-api-access-xdn2q\") pod \"maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.571727 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.571632 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r2kxt\"" Jul 28 17:57:23.579257 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.579222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:23.773333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.773307 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf"] Jul 28 17:57:23.775814 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:57:23.775778 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf8ada0_63db_4c20_a3d8_5c81c3af47b1.slice/crio-aa9bae6019076a00f3f372e3e9383eaba8b37f691cf4f1d953c68dbc85eec1cd WatchSource:0}: Error finding container aa9bae6019076a00f3f372e3e9383eaba8b37f691cf4f1d953c68dbc85eec1cd: Status 404 returned error can't find the container with id aa9bae6019076a00f3f372e3e9383eaba8b37f691cf4f1d953c68dbc85eec1cd Jul 28 17:57:23.777341 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.777322 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:57:23.955137 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:23.954851 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" event={"ID":"adf8ada0-63db-4c20-a3d8-5c81c3af47b1","Type":"ContainerStarted","Data":"aa9bae6019076a00f3f372e3e9383eaba8b37f691cf4f1d953c68dbc85eec1cd"} Jul 28 17:57:24.173723 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:24.173681 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 28 17:57:24.174178 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:24.173743 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 28 17:57:24.253208 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:24.251668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:24.348027 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:24.347986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:25.174020 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:25.173975 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 28 17:57:25.174472 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:25.174051 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 28 17:57:26.177961 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:26.177928 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:26.178425 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:26.178329 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:26.179083 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:26.179060 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 17:57:26.975231 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:26.975141 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" event={"ID":"adf8ada0-63db-4c20-a3d8-5c81c3af47b1","Type":"ContainerStarted","Data":"a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655"} Jul 28 17:57:26.975464 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:26.975449 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:26.994935 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:26.994877 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" podStartSLOduration=1.128211062 podStartE2EDuration="3.994860825s" totalImagesPullingTime="2.866649763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:26.993253224 +0000 UTC m=+2943.796941815" watchObservedRunningTime="2026-07-28 17:57:26.994860825 +0000 UTC m=+2943.798549415" Jul 28 17:57:27.316314 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:27.316221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:31.325564 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:31.325526 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:57:32.986258 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:32.986223 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 17:57:47.789285 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:57:47.789201 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:02.699197 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:02.699155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:06.418485 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:06.418441 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:58:21.336845 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:21.336806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:24.236558 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:24.236530 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:58:24.247413 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:24.247385 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lbjhs_827fb8dd-cbad-4a6a-9b1e-8c2a216cee3f/ovn-acl-logging/0.log" Jul 28 17:58:40.315748 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:40.315705 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:58:50.314188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:50.314145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:52.699891 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:52.699855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:52.991959 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:52.991851 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9"] Jul 28 17:58:52.995615 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:52.995586 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:52.998650 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:52.998623 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-7fdc90-openshift-default-dockercfg-t8d4r\"" Jul 28 17:58:53.013052 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.012990 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9"] Jul 28 17:58:53.027906 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.027872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:53.028900 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.028882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8506e01-05fa-4b55-a65f-89fbc6c66772-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.028995 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.028920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-socket\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.028995 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.028941 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-certs\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.028995 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.028973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxffb\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-kube-api-access-wxffb\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.029145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.029050 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-credential-socket\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.029145 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.029096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-data\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.029259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.029182 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-envoy\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.029259 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.029245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-token\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.029364 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.029300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-podinfo\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130166 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-token\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-podinfo\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8506e01-05fa-4b55-a65f-89fbc6c66772-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-socket\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-certs\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxffb\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-kube-api-access-wxffb\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130366 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-credential-socket\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130700 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-data\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130700 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-envoy\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130811 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-socket\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130811 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130780 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-data\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130905 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130835 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-credential-socket\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.130970 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.130910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-certs\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.131427 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.131407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8506e01-05fa-4b55-a65f-89fbc6c66772-istiod-ca-cert\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.132758 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.132723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-envoy\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.133174 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.133152 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-podinfo\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.139654 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.139607 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-token\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.139791 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.139698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxffb\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-kube-api-access-wxffb\") pod \"e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.306449 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.306349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:53.394744 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.394698 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:53.454971 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.454933 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9"] Jul 28 17:58:53.455855 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:58:53.455826 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8506e01_05fa_4b55_a65f_89fbc6c66772.slice/crio-ced2b73fa0e546b083101ece58d97b615e8d8e02f0a3a856369e77b2351be58e WatchSource:0}: Error finding container ced2b73fa0e546b083101ece58d97b615e8d8e02f0a3a856369e77b2351be58e: Status 404 returned error can't find the container with id ced2b73fa0e546b083101ece58d97b615e8d8e02f0a3a856369e77b2351be58e Jul 28 17:58:53.458372 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.458341 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:58:53.458466 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.458405 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:58:53.458466 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:53.458440 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:58:54.424921 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:54.424883 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" event={"ID":"a8506e01-05fa-4b55-a65f-89fbc6c66772","Type":"ContainerStarted","Data":"8aa11ff9ee641fc92f76c1c41eedbf0723091269ba4e45cc37b3563d79ffaf7d"} Jul 28 17:58:54.425371 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:54.424940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" event={"ID":"a8506e01-05fa-4b55-a65f-89fbc6c66772","Type":"ContainerStarted","Data":"ced2b73fa0e546b083101ece58d97b615e8d8e02f0a3a856369e77b2351be58e"} Jul 28 17:58:54.449165 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:54.449079 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" podStartSLOduration=2.449059554 podStartE2EDuration="2.449059554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:54.448693167 +0000 UTC m=+3031.252381757" watchObservedRunningTime="2026-07-28 17:58:54.449059554 +0000 UTC m=+3031.252748146" Jul 28 17:58:55.306740 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:55.306688 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:55.311984 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:55.311949 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:55.429674 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:55.429640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:55.430709 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:55.430686 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:58:56.453638 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:56.453604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:56.552866 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:56.552828 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:59.698181 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.698144 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth"] Jul 28 17:58:59.709333 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.709297 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.712631 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.712602 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-7fdc90-serving-cert\"" Jul 28 17:58:59.714566 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.714537 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth"] Jul 28 17:58:59.734234 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.734205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:59.792861 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.792816 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.793086 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.792940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d527f980-24a1-473d-9999-b10266f8a48c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.793183 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.793079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdfj\" (UniqueName: \"kubernetes.io/projected/d527f980-24a1-473d-9999-b10266f8a48c-kube-api-access-xxdfj\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.894777 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.894731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d527f980-24a1-473d-9999-b10266f8a48c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.894968 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.894796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdfj\" (UniqueName: \"kubernetes.io/projected/d527f980-24a1-473d-9999-b10266f8a48c-kube-api-access-xxdfj\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.894968 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.894863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.895133 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:58:59.894987 2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-serving-cert: secret "maas-api-e2e-ipp-cleanup-7fdc90-serving-cert" not found Jul 28 17:58:59.895133 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:58:59.895071 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls podName:d527f980-24a1-473d-9999-b10266f8a48c nodeName:}" failed. No retries permitted until 2026-07-28 17:59:00.395046988 +0000 UTC m=+3037.198735563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" (UID: "d527f980-24a1-473d-9999-b10266f8a48c") : secret "maas-api-e2e-ipp-cleanup-7fdc90-serving-cert" not found Jul 28 17:58:59.895437 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.895415 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d527f980-24a1-473d-9999-b10266f8a48c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.905760 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.905725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdfj\" (UniqueName: \"kubernetes.io/projected/d527f980-24a1-473d-9999-b10266f8a48c-kube-api-access-xxdfj\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:58:59.978643 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:58:59.978541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:00.398472 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:00.398426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:59:00.401031 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:00.401011 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:59:00.485238 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:00.485204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:00.580720 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:00.580680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:00.623921 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:00.623884 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:59:00.784296 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:00.784246 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth"] Jul 28 17:59:00.789035 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:59:00.788995 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd527f980_24a1_473d_9999_b10266f8a48c.slice/crio-ecdaa631e12a513aecae72d073e865b601f2095a78a624095357583a258a6b7f WatchSource:0}: Error finding container ecdaa631e12a513aecae72d073e865b601f2095a78a624095357583a258a6b7f: Status 404 returned error can't find the container with id ecdaa631e12a513aecae72d073e865b601f2095a78a624095357583a258a6b7f Jul 28 17:59:01.024261 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:01.024170 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:01.460487 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:01.460435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" event={"ID":"d527f980-24a1-473d-9999-b10266f8a48c","Type":"ContainerStarted","Data":"2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61"} Jul 28 17:59:01.460691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:01.460597 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:59:01.460691 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:01.460642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" event={"ID":"d527f980-24a1-473d-9999-b10266f8a48c","Type":"ContainerStarted","Data":"ecdaa631e12a513aecae72d073e865b601f2095a78a624095357583a258a6b7f"} Jul 28 17:59:01.480067 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:01.480006 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" podStartSLOduration=2.110317092 podStartE2EDuration="2.479988609s" totalImagesPullingTime="369.671517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:01.478952705 +0000 UTC m=+3038.282641297" watchObservedRunningTime="2026-07-28 17:59:01.479988609 +0000 UTC m=+3038.283677199" Jul 28 17:59:07.198970 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:07.198927 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:59:07.472420 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:07.472342 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:59:15.534475 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:15.534441 2579 generic.go:356] "Generic (PLEG): container finished" podID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerID="8aa11ff9ee641fc92f76c1c41eedbf0723091269ba4e45cc37b3563d79ffaf7d" exitCode=0 Jul 28 17:59:15.534900 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:15.534509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" event={"ID":"a8506e01-05fa-4b55-a65f-89fbc6c66772","Type":"ContainerDied","Data":"8aa11ff9ee641fc92f76c1c41eedbf0723091269ba4e45cc37b3563d79ffaf7d"} Jul 28 17:59:15.534946 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:15.534919 2579 scope.go:117] "RemoveContainer" containerID="8aa11ff9ee641fc92f76c1c41eedbf0723091269ba4e45cc37b3563d79ffaf7d" Jul 28 17:59:15.536173 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:15.536143 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:59:15.536239 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:15.536218 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:59:15.536284 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:15.536254 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 28 17:59:16.541187 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:16.541148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" event={"ID":"a8506e01-05fa-4b55-a65f-89fbc6c66772","Type":"ContainerStarted","Data":"27fdb346077cb7c52f91910648cfd7fc7d6083183e9bd1cc74a08ee949f332b5"} Jul 28 17:59:17.096533 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.096493 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6"] Jul 28 17:59:17.099037 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.099017 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" Jul 28 17:59:17.101964 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.101938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 17:59:17.126194 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.126147 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6"] Jul 28 17:59:17.164928 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.164883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6\" (UID: \"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" Jul 28 17:59:17.265780 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.265738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6\" (UID: \"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" Jul 28 17:59:17.266440 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.266418 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6\" (UID: \"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" Jul 28 17:59:17.307191 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.307157 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:59:17.307427 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.307406 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 28 17:59:17.307496 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.307478 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 28 17:59:17.409736 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.409641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" Jul 28 17:59:17.606552 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:17.606518 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6"] Jul 28 17:59:17.607727 ip-10-0-135-73 kubenswrapper[2579]: W0728 17:59:17.607697 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c9048a_2d1a_4c6b_9f80_bdbc53ef2a62.slice/crio-f4d182c89f92ade5c8e824c94860205ac7f3bce955ac4c60ecbdcacfeaf08991 WatchSource:0}: Error finding container f4d182c89f92ade5c8e824c94860205ac7f3bce955ac4c60ecbdcacfeaf08991: Status 404 returned error can't find the container with id f4d182c89f92ade5c8e824c94860205ac7f3bce955ac4c60ecbdcacfeaf08991 Jul 28 17:59:18.307018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:18.306925 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 28 17:59:18.307018 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:18.306980 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 28 17:59:18.563750 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:18.563663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" event={"ID":"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62","Type":"ContainerStarted","Data":"ccd8699e75dd4932d2842e49b781d0d4d31abbf5be220f008929ac832b779554"} Jul 28 17:59:18.563750 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:18.563718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" event={"ID":"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62","Type":"ContainerStarted","Data":"f4d182c89f92ade5c8e824c94860205ac7f3bce955ac4c60ecbdcacfeaf08991"} Jul 28 17:59:18.592483 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:18.592425 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" podStartSLOduration=1.592408422 podStartE2EDuration="1.592408422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:59:18.590911762 +0000 UTC m=+3055.394600355" watchObservedRunningTime="2026-07-28 17:59:18.592408422 +0000 UTC m=+3055.396097011" Jul 28 17:59:19.307753 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:19.307712 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 28 17:59:19.308207 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:19.307775 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 28 17:59:19.570704 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:19.570612 2579 generic.go:356] "Generic (PLEG): container finished" podID="e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62" containerID="ccd8699e75dd4932d2842e49b781d0d4d31abbf5be220f008929ac832b779554" exitCode=0 Jul 28 17:59:19.570704 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:19.570665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" event={"ID":"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62","Type":"ContainerDied","Data":"ccd8699e75dd4932d2842e49b781d0d4d31abbf5be220f008929ac832b779554"} Jul 28 17:59:20.307536 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:20.307495 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 28 17:59:20.307728 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:20.307555 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 28 17:59:20.746423 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:20.746324 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" Jul 28 17:59:20.902968 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:20.902924 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62-maas-api-service-ca\") pod \"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62\" (UID: \"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62\") " Jul 28 17:59:20.903397 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:20.903368 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62" (UID: "e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:21.003757 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:21.003719 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:21.310925 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:21.310841 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:59:21.311163 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:21.311142 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:59:21.312044 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:21.312022 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:59:21.582950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:21.582855 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" Jul 28 17:59:21.582950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:21.582890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6" event={"ID":"e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62","Type":"ContainerDied","Data":"f4d182c89f92ade5c8e824c94860205ac7f3bce955ac4c60ecbdcacfeaf08991"} Jul 28 17:59:21.582950 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:21.582945 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4d182c89f92ade5c8e824c94860205ac7f3bce955ac4c60ecbdcacfeaf08991" Jul 28 17:59:31.015670 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.015628 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth"] Jul 28 17:59:31.016262 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.015844 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" podUID="d527f980-24a1-473d-9999-b10266f8a48c" containerName="maas-api" containerID="cri-o://2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61" gracePeriod=30 Jul 28 17:59:31.050582 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.050531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:31.209399 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.209364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:31.282460 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.282431 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:59:31.307712 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.307681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:31.400886 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.400847 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d527f980-24a1-473d-9999-b10266f8a48c-openshift-service-ca\") pod \"d527f980-24a1-473d-9999-b10266f8a48c\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " Jul 28 17:59:31.401082 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.400966 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxdfj\" (UniqueName: \"kubernetes.io/projected/d527f980-24a1-473d-9999-b10266f8a48c-kube-api-access-xxdfj\") pod \"d527f980-24a1-473d-9999-b10266f8a48c\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " Jul 28 17:59:31.401082 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.400992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls\") pod \"d527f980-24a1-473d-9999-b10266f8a48c\" (UID: \"d527f980-24a1-473d-9999-b10266f8a48c\") " Jul 28 17:59:31.401252 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.401228 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d527f980-24a1-473d-9999-b10266f8a48c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d527f980-24a1-473d-9999-b10266f8a48c" (UID: "d527f980-24a1-473d-9999-b10266f8a48c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:31.403227 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.403197 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d527f980-24a1-473d-9999-b10266f8a48c" (UID: "d527f980-24a1-473d-9999-b10266f8a48c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:59:31.403355 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.403255 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d527f980-24a1-473d-9999-b10266f8a48c-kube-api-access-xxdfj" (OuterVolumeSpecName: "kube-api-access-xxdfj") pod "d527f980-24a1-473d-9999-b10266f8a48c" (UID: "d527f980-24a1-473d-9999-b10266f8a48c"). InnerVolumeSpecName "kube-api-access-xxdfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:59:31.502572 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.502532 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d527f980-24a1-473d-9999-b10266f8a48c-openshift-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:31.502572 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.502563 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdfj\" (UniqueName: \"kubernetes.io/projected/d527f980-24a1-473d-9999-b10266f8a48c-kube-api-access-xxdfj\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:31.502572 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.502573 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d527f980-24a1-473d-9999-b10266f8a48c-maas-api-tls\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:31.633286 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.633254 2579 generic.go:356] "Generic (PLEG): container finished" podID="d527f980-24a1-473d-9999-b10266f8a48c" containerID="2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61" exitCode=0 Jul 28 17:59:31.633471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.633298 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" Jul 28 17:59:31.633471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.633341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" event={"ID":"d527f980-24a1-473d-9999-b10266f8a48c","Type":"ContainerDied","Data":"2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61"} Jul 28 17:59:31.633471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.633393 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth" event={"ID":"d527f980-24a1-473d-9999-b10266f8a48c","Type":"ContainerDied","Data":"ecdaa631e12a513aecae72d073e865b601f2095a78a624095357583a258a6b7f"} Jul 28 17:59:31.633471 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.633416 2579 scope.go:117] "RemoveContainer" containerID="2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61" Jul 28 17:59:31.643706 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.643686 2579 scope.go:117] "RemoveContainer" containerID="2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61" Jul 28 17:59:31.643969 ip-10-0-135-73 kubenswrapper[2579]: E0728 17:59:31.643949 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61\": container with ID starting with 2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61 not found: ID does not exist" containerID="2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61" Jul 28 17:59:31.644021 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.643979 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61"} err="failed to get container status \"2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61\": rpc error: code = NotFound desc = could not find container \"2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61\": container with ID starting with 2c05a4ae229750e505cc641a684a5ed4c238aad1026f7f0d8cf58723277a7b61 not found: ID does not exist" Jul 28 17:59:31.658956 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.658922 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth"] Jul 28 17:59:31.667671 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.667641 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-7fdc90-66cd5b8cbc-g5nth"] Jul 28 17:59:31.684483 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:31.684457 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d527f980-24a1-473d-9999-b10266f8a48c" path="/var/lib/kubelet/pods/d527f980-24a1-473d-9999-b10266f8a48c/volumes" Jul 28 17:59:32.184367 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:32.184312 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6"] Jul 28 17:59:32.187282 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:32.187258 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-7fdc90-fkgk6"] Jul 28 17:59:32.187411 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:32.187392 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:32.603857 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:32.603823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:33.684406 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:33.684370 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62" path="/var/lib/kubelet/pods/e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62/volumes" Jul 28 17:59:36.003220 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:36.003179 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 17:59:39.230076 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:39.230009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:39.330639 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:39.330599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:39.495463 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:39.495367 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9"] Jul 28 17:59:39.495658 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:39.495615 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" containerID="cri-o://27fdb346077cb7c52f91910648cfd7fc7d6083183e9bd1cc74a08ee949f332b5" gracePeriod=30 Jul 28 17:59:39.524788 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:39.524748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:39.804305 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:39.804210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:44.707606 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.707567 2579 generic.go:356] "Generic (PLEG): container finished" podID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerID="27fdb346077cb7c52f91910648cfd7fc7d6083183e9bd1cc74a08ee949f332b5" exitCode=0 Jul 28 17:59:44.708152 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.707643 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" event={"ID":"a8506e01-05fa-4b55-a65f-89fbc6c66772","Type":"ContainerDied","Data":"27fdb346077cb7c52f91910648cfd7fc7d6083183e9bd1cc74a08ee949f332b5"} Jul 28 17:59:44.708152 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.707732 2579 scope.go:117] "RemoveContainer" containerID="8aa11ff9ee641fc92f76c1c41eedbf0723091269ba4e45cc37b3563d79ffaf7d" Jul 28 17:59:44.765859 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.765781 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:59:44.829081 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829040 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-podinfo\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829280 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829091 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-socket\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829280 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829129 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8506e01-05fa-4b55-a65f-89fbc6c66772-istiod-ca-cert\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829280 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829148 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-envoy\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829441 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-certs\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829441 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829386 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxffb\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-kube-api-access-wxffb\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829441 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:59:44.829600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829455 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-credential-socket\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829480 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-data\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829511 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-token\") pod \"a8506e01-05fa-4b55-a65f-89fbc6c66772\" (UID: \"a8506e01-05fa-4b55-a65f-89fbc6c66772\") " Jul 28 17:59:44.829600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829569 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8506e01-05fa-4b55-a65f-89fbc6c66772-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:44.829600 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829579 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:59:44.829878 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829701 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:59:44.829878 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829828 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.829878 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829849 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8506e01-05fa-4b55-a65f-89fbc6c66772-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.829878 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829864 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.829878 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.829876 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.830407 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.830386 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-data" (OuterVolumeSpecName: "istio-data") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:59:44.831387 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.831362 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 17:59:44.831499 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.831476 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-kube-api-access-wxffb" (OuterVolumeSpecName: "kube-api-access-wxffb") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "kube-api-access-wxffb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:59:44.831767 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.831739 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:59:44.831767 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.831759 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-token" (OuterVolumeSpecName: "istio-token") pod "a8506e01-05fa-4b55-a65f-89fbc6c66772" (UID: "a8506e01-05fa-4b55-a65f-89fbc6c66772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:59:44.931188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.931141 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.931188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.931179 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxffb\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-kube-api-access-wxffb\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.931188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.931190 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.931188 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.931198 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:44.931480 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:44.931206 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8506e01-05fa-4b55-a65f-89fbc6c66772-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 17:59:45.713401 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:45.713369 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" Jul 28 17:59:45.713839 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:45.713413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9" event={"ID":"a8506e01-05fa-4b55-a65f-89fbc6c66772","Type":"ContainerDied","Data":"ced2b73fa0e546b083101ece58d97b615e8d8e02f0a3a856369e77b2351be58e"} Jul 28 17:59:45.713839 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:45.713470 2579 scope.go:117] "RemoveContainer" containerID="27fdb346077cb7c52f91910648cfd7fc7d6083183e9bd1cc74a08ee949f332b5" Jul 28 17:59:45.735398 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:45.735367 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9"] Jul 28 17:59:45.740304 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:45.740276 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-7fdc90-openshift-default-5b4994978b-cxrt9"] Jul 28 17:59:47.684721 ip-10-0-135-73 kubenswrapper[2579]: I0728 17:59:47.684686 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" path="/var/lib/kubelet/pods/a8506e01-05fa-4b55-a65f-89fbc6c66772/volumes" Jul 28 18:00:00.186877 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.186835 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg"] Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187172 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187184 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187193 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d527f980-24a1-473d-9999-b10266f8a48c" containerName="maas-api" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187199 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d527f980-24a1-473d-9999-b10266f8a48c" containerName="maas-api" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187206 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187213 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187221 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62" containerName="revoke-keys" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187228 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62" containerName="revoke-keys" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187287 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c9048a-2d1a-4c6b-9f80-bdbc53ef2a62" containerName="revoke-keys" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187297 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187303 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d527f980-24a1-473d-9999-b10266f8a48c" containerName="maas-api" Jul 28 18:00:00.187304 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.187308 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8506e01-05fa-4b55-a65f-89fbc6c66772" containerName="istio-proxy" Jul 28 18:00:00.191908 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.191886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg" Jul 28 18:00:00.199180 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.199154 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h"] Jul 28 18:00:00.202464 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.202446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" Jul 28 18:00:00.205475 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.205449 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 18:00:00.211578 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.211553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" Jul 28 18:00:00.212372 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.212354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg" Jul 28 18:00:00.219208 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.219181 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg"] Jul 28 18:00:00.224030 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.223994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h"] Jul 28 18:00:00.238839 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.238784 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml"] Jul 28 18:00:00.244028 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.243987 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd"] Jul 28 18:00:00.244244 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.244151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml" Jul 28 18:00:00.247916 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.247888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd" Jul 28 18:00:00.262786 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.262757 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml" Jul 28 18:00:00.271899 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.271815 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml"] Jul 28 18:00:00.274451 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.274415 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p"] Jul 28 18:00:00.277752 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.277726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd" Jul 28 18:00:00.286041 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.282068 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p" Jul 28 18:00:00.293269 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.293236 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd"] Jul 28 18:00:00.297343 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.297309 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p"] Jul 28 18:00:00.327323 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.326568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 18:00:00.329605 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.329412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p" Jul 28 18:00:00.471456 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.471320 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h"] Jul 28 18:00:00.475689 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:00:00.475639 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56bcbed7_d878_46df_9a9f_e50b51514fb7.slice/crio-39c930d64383c3c82e1d3a6bc7a7255e7855d446ec398e03c2ab33866130fa45 WatchSource:0}: Error finding container 39c930d64383c3c82e1d3a6bc7a7255e7855d446ec398e03c2ab33866130fa45: Status 404 returned error can't find the container with id 39c930d64383c3c82e1d3a6bc7a7255e7855d446ec398e03c2ab33866130fa45 Jul 28 18:00:00.507214 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.507178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg"] Jul 28 18:00:00.522170 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:00:00.522078 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f607985_19eb_46e0_829a_d203b8545a08.slice/crio-864fb24e649738710e27c1ec180936e8f04843ae8018a84cd5548edabe713e0d WatchSource:0}: Error finding container 864fb24e649738710e27c1ec180936e8f04843ae8018a84cd5548edabe713e0d: Status 404 returned error can't find the container with id 864fb24e649738710e27c1ec180936e8f04843ae8018a84cd5548edabe713e0d Jul 28 18:00:00.540463 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.540389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml"] Jul 28 18:00:00.541551 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:00:00.541522 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d538d88_60f3_4298_b542_c1f048305815.slice/crio-5024b783a7c3cef06ce5774c711e7b4fdb25c2bcb8f1902d33c94b34e9365777 WatchSource:0}: Error finding container 5024b783a7c3cef06ce5774c711e7b4fdb25c2bcb8f1902d33c94b34e9365777: Status 404 returned error can't find the container with id 5024b783a7c3cef06ce5774c711e7b4fdb25c2bcb8f1902d33c94b34e9365777 Jul 28 18:00:00.559554 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:00:00.559522 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500efae2_0aec_4877_ba56_d52eb8341e16.slice/crio-8af08da466894953bfe7342bfb12e31ff6aed8aa0639d53a256f847e731be736 WatchSource:0}: Error finding container 8af08da466894953bfe7342bfb12e31ff6aed8aa0639d53a256f847e731be736: Status 404 returned error can't find the container with id 8af08da466894953bfe7342bfb12e31ff6aed8aa0639d53a256f847e731be736 Jul 28 18:00:00.560560 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.560538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd"] Jul 28 18:00:00.577861 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.577834 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p"] Jul 28 18:00:00.579537 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:00:00.579504 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0449f0d7_5d54_4d2a_9621_547cd1cc767f.slice/crio-aa083fdae5ccb589955b80b995284caa0bb2fe2c7bb4e1743a7508df87bff30d WatchSource:0}: Error finding container aa083fdae5ccb589955b80b995284caa0bb2fe2c7bb4e1743a7508df87bff30d: Status 404 returned error can't find the container with id aa083fdae5ccb589955b80b995284caa0bb2fe2c7bb4e1743a7508df87bff30d Jul 28 18:00:00.792224 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.792184 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p" event={"ID":"0449f0d7-5d54-4d2a-9621-547cd1cc767f","Type":"ContainerStarted","Data":"aa083fdae5ccb589955b80b995284caa0bb2fe2c7bb4e1743a7508df87bff30d"} Jul 28 18:00:00.793387 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.793361 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml" event={"ID":"7d538d88-60f3-4298-b542-c1f048305815","Type":"ContainerStarted","Data":"5024b783a7c3cef06ce5774c711e7b4fdb25c2bcb8f1902d33c94b34e9365777"} Jul 28 18:00:00.794551 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.794527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg" event={"ID":"8f607985-19eb-46e0-829a-d203b8545a08","Type":"ContainerStarted","Data":"864fb24e649738710e27c1ec180936e8f04843ae8018a84cd5548edabe713e0d"} Jul 28 18:00:00.795958 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.795932 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" event={"ID":"56bcbed7-d878-46df-9a9f-e50b51514fb7","Type":"ContainerStarted","Data":"b8ba809ef0992f02f8e0bf5b45b8459e7122d6e3f07f77146aa4a5210f6265c6"} Jul 28 18:00:00.796069 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.795978 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" event={"ID":"56bcbed7-d878-46df-9a9f-e50b51514fb7","Type":"ContainerStarted","Data":"39c930d64383c3c82e1d3a6bc7a7255e7855d446ec398e03c2ab33866130fa45"} Jul 28 18:00:00.797154 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.797132 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd" event={"ID":"500efae2-0aec-4877-ba56-d52eb8341e16","Type":"ContainerStarted","Data":"8af08da466894953bfe7342bfb12e31ff6aed8aa0639d53a256f847e731be736"} Jul 28 18:00:00.814356 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:00.814301 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" podStartSLOduration=0.814284184 podStartE2EDuration="814.284184ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:00.812975347 +0000 UTC m=+3097.616663925" watchObservedRunningTime="2026-07-28 18:00:00.814284184 +0000 UTC m=+3097.617972774" Jul 28 18:00:01.805072 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.805017 2579 generic.go:356] "Generic (PLEG): container finished" podID="500efae2-0aec-4877-ba56-d52eb8341e16" containerID="50242a2e7ea552e34828fbc5f6f27db8d50acb4fba4eea9d60c129d0bbedbeca" exitCode=0 Jul 28 18:00:01.805490 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.805103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd" event={"ID":"500efae2-0aec-4877-ba56-d52eb8341e16","Type":"ContainerDied","Data":"50242a2e7ea552e34828fbc5f6f27db8d50acb4fba4eea9d60c129d0bbedbeca"} Jul 28 18:00:01.806734 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.806709 2579 generic.go:356] "Generic (PLEG): container finished" podID="0449f0d7-5d54-4d2a-9621-547cd1cc767f" containerID="1f1525c3ceb6ebcdc01cd966977d710e53ce9269b456a0fe23c8882ab71f91dc" exitCode=0 Jul 28 18:00:01.806831 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.806795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p" event={"ID":"0449f0d7-5d54-4d2a-9621-547cd1cc767f","Type":"ContainerDied","Data":"1f1525c3ceb6ebcdc01cd966977d710e53ce9269b456a0fe23c8882ab71f91dc"} Jul 28 18:00:01.808135 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.808096 2579 generic.go:356] "Generic (PLEG): container finished" podID="7d538d88-60f3-4298-b542-c1f048305815" containerID="a1e2b9f68e413f46bb749b8551840188db1c63bef0b14019513c1164da0518e0" exitCode=0 Jul 28 18:00:01.808232 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.808187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml" event={"ID":"7d538d88-60f3-4298-b542-c1f048305815","Type":"ContainerDied","Data":"a1e2b9f68e413f46bb749b8551840188db1c63bef0b14019513c1164da0518e0"} Jul 28 18:00:01.809731 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.809712 2579 generic.go:356] "Generic (PLEG): container finished" podID="8f607985-19eb-46e0-829a-d203b8545a08" containerID="8741c59c1265fb9626f3d0766fa7b1d189f6baf890cc29c7d005d3a540a5bf9c" exitCode=0 Jul 28 18:00:01.809845 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.809736 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg" event={"ID":"8f607985-19eb-46e0-829a-d203b8545a08","Type":"ContainerDied","Data":"8741c59c1265fb9626f3d0766fa7b1d189f6baf890cc29c7d005d3a540a5bf9c"} Jul 28 18:00:01.811177 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.811157 2579 generic.go:356] "Generic (PLEG): container finished" podID="56bcbed7-d878-46df-9a9f-e50b51514fb7" containerID="b8ba809ef0992f02f8e0bf5b45b8459e7122d6e3f07f77146aa4a5210f6265c6" exitCode=0 Jul 28 18:00:01.811263 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:01.811201 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" event={"ID":"56bcbed7-d878-46df-9a9f-e50b51514fb7","Type":"ContainerDied","Data":"b8ba809ef0992f02f8e0bf5b45b8459e7122d6e3f07f77146aa4a5210f6265c6"} Jul 28 18:00:02.807058 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:02.807024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:02.842987 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:02.842894 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml"] Jul 28 18:00:02.938536 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:02.938503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:03.022225 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.022163 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd" Jul 28 18:00:03.040345 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.040315 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p" Jul 28 18:00:03.042355 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.042323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:03.045047 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.045024 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" Jul 28 18:00:03.050676 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.050645 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg" Jul 28 18:00:03.055068 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.055039 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml" Jul 28 18:00:03.455584 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.455542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:03.826620 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.826588 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd" Jul 28 18:00:03.827065 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.826627 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd" event={"ID":"500efae2-0aec-4877-ba56-d52eb8341e16","Type":"ContainerDied","Data":"8af08da466894953bfe7342bfb12e31ff6aed8aa0639d53a256f847e731be736"} Jul 28 18:00:03.827065 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.826711 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af08da466894953bfe7342bfb12e31ff6aed8aa0639d53a256f847e731be736" Jul 28 18:00:03.828091 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.828058 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p" event={"ID":"0449f0d7-5d54-4d2a-9621-547cd1cc767f","Type":"ContainerDied","Data":"aa083fdae5ccb589955b80b995284caa0bb2fe2c7bb4e1743a7508df87bff30d"} Jul 28 18:00:03.828239 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.828135 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa083fdae5ccb589955b80b995284caa0bb2fe2c7bb4e1743a7508df87bff30d" Jul 28 18:00:03.828239 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.828068 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754360-zhn6p" Jul 28 18:00:03.829496 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.829474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml" event={"ID":"7d538d88-60f3-4298-b542-c1f048305815","Type":"ContainerDied","Data":"5024b783a7c3cef06ce5774c711e7b4fdb25c2bcb8f1902d33c94b34e9365777"} Jul 28 18:00:03.829496 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.829496 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml" Jul 28 18:00:03.829647 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.829507 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5024b783a7c3cef06ce5774c711e7b4fdb25c2bcb8f1902d33c94b34e9365777" Jul 28 18:00:03.831018 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.831000 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg" Jul 28 18:00:03.831208 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.830996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg" event={"ID":"8f607985-19eb-46e0-829a-d203b8545a08","Type":"ContainerDied","Data":"864fb24e649738710e27c1ec180936e8f04843ae8018a84cd5548edabe713e0d"} Jul 28 18:00:03.831208 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.831127 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="864fb24e649738710e27c1ec180936e8f04843ae8018a84cd5548edabe713e0d" Jul 28 18:00:03.832459 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.832441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" event={"ID":"56bcbed7-d878-46df-9a9f-e50b51514fb7","Type":"ContainerDied","Data":"39c930d64383c3c82e1d3a6bc7a7255e7855d446ec398e03c2ab33866130fa45"} Jul 28 18:00:03.832552 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.832465 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h" Jul 28 18:00:03.832552 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.832478 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39c930d64383c3c82e1d3a6bc7a7255e7855d446ec398e03c2ab33866130fa45" Jul 28 18:00:03.867217 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.867179 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml"] Jul 28 18:00:03.870290 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.870258 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-20ce8c-29754360-kmgml"] Jul 28 18:00:03.899408 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:03.899376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:05.685685 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:05.685644 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d538d88-60f3-4298-b542-c1f048305815" path="/var/lib/kubelet/pods/7d538d88-60f3-4298-b542-c1f048305815/volumes" Jul 28 18:00:06.536414 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536372 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xlwzk"] Jul 28 18:00:06.536732 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536718 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0449f0d7-5d54-4d2a-9621-547cd1cc767f" containerName="cleanup" Jul 28 18:00:06.536780 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536736 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0449f0d7-5d54-4d2a-9621-547cd1cc767f" containerName="cleanup" Jul 28 18:00:06.536780 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536750 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56bcbed7-d878-46df-9a9f-e50b51514fb7" containerName="cleanup" Jul 28 18:00:06.536780 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536757 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56bcbed7-d878-46df-9a9f-e50b51514fb7" containerName="cleanup" Jul 28 18:00:06.536780 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536768 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d538d88-60f3-4298-b542-c1f048305815" containerName="cleanup" Jul 28 18:00:06.536780 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536775 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d538d88-60f3-4298-b542-c1f048305815" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536784 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f607985-19eb-46e0-829a-d203b8545a08" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536790 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f607985-19eb-46e0-829a-d203b8545a08" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536796 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="500efae2-0aec-4877-ba56-d52eb8341e16" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536801 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="500efae2-0aec-4877-ba56-d52eb8341e16" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536871 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="56bcbed7-d878-46df-9a9f-e50b51514fb7" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536878 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0449f0d7-5d54-4d2a-9621-547cd1cc767f" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536885 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="500efae2-0aec-4877-ba56-d52eb8341e16" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536890 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d538d88-60f3-4298-b542-c1f048305815" containerName="cleanup" Jul 28 18:00:06.536945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.536895 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f607985-19eb-46e0-829a-d203b8545a08" containerName="cleanup" Jul 28 18:00:06.540124 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.540085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:06.542854 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.542831 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-6vxgj\"" Jul 28 18:00:06.549327 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.549300 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xlwzk"] Jul 28 18:00:06.615635 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.615590 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8xg\" (UniqueName: \"kubernetes.io/projected/8ded6712-6ff8-4786-a11c-9b3857456a66-kube-api-access-kg8xg\") pod \"kuadrant-operator-catalog-xlwzk\" (UID: \"8ded6712-6ff8-4786-a11c-9b3857456a66\") " pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:06.717143 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.717068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8xg\" (UniqueName: \"kubernetes.io/projected/8ded6712-6ff8-4786-a11c-9b3857456a66-kube-api-access-kg8xg\") pod \"kuadrant-operator-catalog-xlwzk\" (UID: \"8ded6712-6ff8-4786-a11c-9b3857456a66\") " pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:06.727846 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.727815 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8xg\" (UniqueName: \"kubernetes.io/projected/8ded6712-6ff8-4786-a11c-9b3857456a66-kube-api-access-kg8xg\") pod \"kuadrant-operator-catalog-xlwzk\" (UID: \"8ded6712-6ff8-4786-a11c-9b3857456a66\") " pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:06.850026 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.849989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:06.993773 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:06.993732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xlwzk"] Jul 28 18:00:06.996433 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:00:06.996395 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ded6712_6ff8_4786_a11c_9b3857456a66.slice/crio-c6d0eb2731c2d99aa6578fabea0fa492d5f622e5456410c62a2872cca80eb811 WatchSource:0}: Error finding container c6d0eb2731c2d99aa6578fabea0fa492d5f622e5456410c62a2872cca80eb811: Status 404 returned error can't find the container with id c6d0eb2731c2d99aa6578fabea0fa492d5f622e5456410c62a2872cca80eb811 Jul 28 18:00:07.856753 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:07.856714 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" event={"ID":"8ded6712-6ff8-4786-a11c-9b3857456a66","Type":"ContainerStarted","Data":"c6d0eb2731c2d99aa6578fabea0fa492d5f622e5456410c62a2872cca80eb811"} Jul 28 18:00:08.969654 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:08.969618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:09.870791 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:09.870737 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" event={"ID":"8ded6712-6ff8-4786-a11c-9b3857456a66","Type":"ContainerStarted","Data":"6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd"} Jul 28 18:00:09.911869 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:09.911810 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" podStartSLOduration=1.748396245 podStartE2EDuration="3.911790656s" totalImagesPullingTime="2.163394411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:09.910377274 +0000 UTC m=+3106.714065864" watchObservedRunningTime="2026-07-28 18:00:09.911790656 +0000 UTC m=+3106.715479246" Jul 28 18:00:13.803374 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:13.803332 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 18:00:16.850841 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:16.850796 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:16.851344 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:16.850866 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:16.872745 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:16.872717 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:16.928514 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:16.928482 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:17.123225 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:17.123127 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xlwzk"] Jul 28 18:00:18.917305 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:18.917251 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" podUID="8ded6712-6ff8-4786-a11c-9b3857456a66" containerName="registry-server" containerID="cri-o://6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd" gracePeriod=2 Jul 28 18:00:19.166684 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.166658 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:19.343629 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.343583 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kg8xg\" (UniqueName: \"kubernetes.io/projected/8ded6712-6ff8-4786-a11c-9b3857456a66-kube-api-access-kg8xg\") pod \"8ded6712-6ff8-4786-a11c-9b3857456a66\" (UID: \"8ded6712-6ff8-4786-a11c-9b3857456a66\") " Jul 28 18:00:19.345700 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.345668 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ded6712-6ff8-4786-a11c-9b3857456a66-kube-api-access-kg8xg" (OuterVolumeSpecName: "kube-api-access-kg8xg") pod "8ded6712-6ff8-4786-a11c-9b3857456a66" (UID: "8ded6712-6ff8-4786-a11c-9b3857456a66"). InnerVolumeSpecName "kube-api-access-kg8xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:00:19.445078 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.445038 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg8xg\" (UniqueName: \"kubernetes.io/projected/8ded6712-6ff8-4786-a11c-9b3857456a66-kube-api-access-kg8xg\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:19.923841 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.923802 2579 generic.go:356] "Generic (PLEG): container finished" podID="8ded6712-6ff8-4786-a11c-9b3857456a66" containerID="6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd" exitCode=0 Jul 28 18:00:19.924302 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.923850 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" Jul 28 18:00:19.924302 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.923882 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" event={"ID":"8ded6712-6ff8-4786-a11c-9b3857456a66","Type":"ContainerDied","Data":"6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd"} Jul 28 18:00:19.924302 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.923933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xlwzk" event={"ID":"8ded6712-6ff8-4786-a11c-9b3857456a66","Type":"ContainerDied","Data":"c6d0eb2731c2d99aa6578fabea0fa492d5f622e5456410c62a2872cca80eb811"} Jul 28 18:00:19.924302 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.923952 2579 scope.go:117] "RemoveContainer" containerID="6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd" Jul 28 18:00:19.933582 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.933562 2579 scope.go:117] "RemoveContainer" containerID="6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd" Jul 28 18:00:19.933907 ip-10-0-135-73 kubenswrapper[2579]: E0728 18:00:19.933887 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd\": container with ID starting with 6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd not found: ID does not exist" containerID="6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd" Jul 28 18:00:19.933987 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.933917 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd"} err="failed to get container status \"6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd\": rpc error: code = NotFound desc = could not find container \"6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd\": container with ID starting with 6c8a875ef6e6b80c559b4195313537569728e5e263a44141bc6292d9ce4e1ccd not found: ID does not exist" Jul 28 18:00:19.953671 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.953583 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xlwzk"] Jul 28 18:00:19.960029 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:19.959996 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xlwzk"] Jul 28 18:00:21.685182 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:21.685150 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ded6712-6ff8-4786-a11c-9b3857456a66" path="/var/lib/kubelet/pods/8ded6712-6ff8-4786-a11c-9b3857456a66/volumes" Jul 28 18:00:33.832756 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:33.832715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:33.938310 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:33.938272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:46.540795 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:46.540704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:46.898830 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:46.898787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:53.548815 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:53.548778 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:53.648997 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:53.648953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:53.851863 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:53.851827 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd"] Jul 28 18:00:53.852073 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:53.852028 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" containerID="cri-o://c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86" gracePeriod=30 Jul 28 18:00:53.869946 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:53.869909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:54.146267 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.146168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:54.301785 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.301746 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf"] Jul 28 18:00:54.302090 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.302077 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ded6712-6ff8-4786-a11c-9b3857456a66" containerName="registry-server" Jul 28 18:00:54.302164 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.302091 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ded6712-6ff8-4786-a11c-9b3857456a66" containerName="registry-server" Jul 28 18:00:54.302203 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.302178 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ded6712-6ff8-4786-a11c-9b3857456a66" containerName="registry-server" Jul 28 18:00:54.309455 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.309428 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" Jul 28 18:00:54.314031 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.314005 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 18:00:54.316831 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.316801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf"] Jul 28 18:00:54.464603 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.464501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f8d924-cdd2-44eb-9b13-fd2ead803d49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf\" (UID: \"12f8d924-cdd2-44eb-9b13-fd2ead803d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" Jul 28 18:00:54.565270 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.565234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f8d924-cdd2-44eb-9b13-fd2ead803d49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf\" (UID: \"12f8d924-cdd2-44eb-9b13-fd2ead803d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" Jul 28 18:00:54.565869 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.565847 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f8d924-cdd2-44eb-9b13-fd2ead803d49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf\" (UID: \"12f8d924-cdd2-44eb-9b13-fd2ead803d49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" Jul 28 18:00:54.621791 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.621743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" Jul 28 18:00:54.768281 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:54.768254 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf"] Jul 28 18:00:54.770545 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:00:54.770515 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f8d924_cdd2_44eb_9b13_fd2ead803d49.slice/crio-c268eb8ad66a18c938100952378aae5ba09071e868684381989b2136cffea769 WatchSource:0}: Error finding container c268eb8ad66a18c938100952378aae5ba09071e868684381989b2136cffea769: Status 404 returned error can't find the container with id c268eb8ad66a18c938100952378aae5ba09071e868684381989b2136cffea769 Jul 28 18:00:55.117021 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:55.116979 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" event={"ID":"12f8d924-cdd2-44eb-9b13-fd2ead803d49","Type":"ContainerStarted","Data":"18e4ec0eab0724b409834fc9a68fd7bd37ffa5818976a19c21109cdeb4589d80"} Jul 28 18:00:55.117232 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:55.117075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" event={"ID":"12f8d924-cdd2-44eb-9b13-fd2ead803d49","Type":"ContainerStarted","Data":"c268eb8ad66a18c938100952378aae5ba09071e868684381989b2136cffea769"} Jul 28 18:00:55.150530 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:55.150467 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" podStartSLOduration=1.150449493 podStartE2EDuration="1.150449493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:55.149024469 +0000 UTC m=+3151.952713060" watchObservedRunningTime="2026-07-28 18:00:55.150449493 +0000 UTC m=+3151.954138131" Jul 28 18:00:56.124140 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:56.124091 2579 generic.go:356] "Generic (PLEG): container finished" podID="12f8d924-cdd2-44eb-9b13-fd2ead803d49" containerID="18e4ec0eab0724b409834fc9a68fd7bd37ffa5818976a19c21109cdeb4589d80" exitCode=0 Jul 28 18:00:56.124545 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:56.124145 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" event={"ID":"12f8d924-cdd2-44eb-9b13-fd2ead803d49","Type":"ContainerDied","Data":"18e4ec0eab0724b409834fc9a68fd7bd37ffa5818976a19c21109cdeb4589d80"} Jul 28 18:00:56.181493 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:56.181448 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 18:00:56.181680 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:56.181513 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:00:57.264164 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:57.264133 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" Jul 28 18:00:57.393668 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:57.393559 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f8d924-cdd2-44eb-9b13-fd2ead803d49-maas-api-service-ca\") pod \"12f8d924-cdd2-44eb-9b13-fd2ead803d49\" (UID: \"12f8d924-cdd2-44eb-9b13-fd2ead803d49\") " Jul 28 18:00:57.393955 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:57.393930 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f8d924-cdd2-44eb-9b13-fd2ead803d49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12f8d924-cdd2-44eb-9b13-fd2ead803d49" (UID: "12f8d924-cdd2-44eb-9b13-fd2ead803d49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:57.494992 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:57.494950 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12f8d924-cdd2-44eb-9b13-fd2ead803d49-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:58.136210 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:58.136178 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" Jul 28 18:00:58.136393 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:58.136218 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf" event={"ID":"12f8d924-cdd2-44eb-9b13-fd2ead803d49","Type":"ContainerDied","Data":"c268eb8ad66a18c938100952378aae5ba09071e868684381989b2136cffea769"} Jul 28 18:00:58.136393 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:58.136264 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c268eb8ad66a18c938100952378aae5ba09071e868684381989b2136cffea769" Jul 28 18:00:59.109282 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.109252 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 18:00:59.146254 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.146177 2579 generic.go:356] "Generic (PLEG): container finished" podID="357b85d9-1e44-4ab2-bdad-213169862e53" containerID="c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86" exitCode=0 Jul 28 18:00:59.146254 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.146224 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" Jul 28 18:00:59.146432 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.146261 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" event={"ID":"357b85d9-1e44-4ab2-bdad-213169862e53","Type":"ContainerDied","Data":"c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86"} Jul 28 18:00:59.146432 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.146328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd" event={"ID":"357b85d9-1e44-4ab2-bdad-213169862e53","Type":"ContainerDied","Data":"83f12555baf2c8077ffd9bfcbd25749e6a2f68daf4ea214cf8e2f3295f2038be"} Jul 28 18:00:59.146432 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.146366 2579 scope.go:117] "RemoveContainer" containerID="c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86" Jul 28 18:00:59.157438 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.157415 2579 scope.go:117] "RemoveContainer" containerID="998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849" Jul 28 18:00:59.166401 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.166380 2579 scope.go:117] "RemoveContainer" containerID="c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86" Jul 28 18:00:59.166691 ip-10-0-135-73 kubenswrapper[2579]: E0728 18:00:59.166671 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86\": container with ID starting with c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86 not found: ID does not exist" containerID="c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86" Jul 28 18:00:59.166764 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.166699 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86"} err="failed to get container status \"c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86\": rpc error: code = NotFound desc = could not find container \"c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86\": container with ID starting with c0efdfba18867ec80176213beb91ddf27ba9f3598c7aae19ffb4cc84a52f7e86 not found: ID does not exist" Jul 28 18:00:59.166764 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.166719 2579 scope.go:117] "RemoveContainer" containerID="998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849" Jul 28 18:00:59.166982 ip-10-0-135-73 kubenswrapper[2579]: E0728 18:00:59.166964 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849\": container with ID starting with 998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849 not found: ID does not exist" containerID="998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849" Jul 28 18:00:59.167035 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.166987 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849"} err="failed to get container status \"998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849\": rpc error: code = NotFound desc = could not find container \"998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849\": container with ID starting with 998d62c93b38d6af12f03dd5fb0b04e0935d93427f18b4e60f5647225369a849 not found: ID does not exist" Jul 28 18:00:59.210236 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210192 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-credential-socket\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210236 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210241 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-envoy\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210492 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210286 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b85d9-1e44-4ab2-bdad-213169862e53-istio-podinfo\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210492 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-data\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210492 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-socket\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210492 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210361 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-istio-token\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210492 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9mjp\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-kube-api-access-k9mjp\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210492 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-certs\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210821 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210540 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:00:59.210821 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210549 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b85d9-1e44-4ab2-bdad-213169862e53-istiod-ca-cert\") pod \"357b85d9-1e44-4ab2-bdad-213169862e53\" (UID: \"357b85d9-1e44-4ab2-bdad-213169862e53\") " Jul 28 18:00:59.210821 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210673 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:00:59.210975 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:00:59.210975 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210888 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.210975 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.210906 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.211172 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.211131 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357b85d9-1e44-4ab2-bdad-213169862e53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:59.211234 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.211179 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-data" (OuterVolumeSpecName: "istio-data") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:00:59.212666 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.212643 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/357b85d9-1e44-4ab2-bdad-213169862e53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:00:59.212807 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.212699 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-kube-api-access-k9mjp" (OuterVolumeSpecName: "kube-api-access-k9mjp") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "kube-api-access-k9mjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:00:59.212988 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.212947 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-istio-token" (OuterVolumeSpecName: "istio-token") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:00:59.212988 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.212964 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "357b85d9-1e44-4ab2-bdad-213169862e53" (UID: "357b85d9-1e44-4ab2-bdad-213169862e53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:00:59.311618 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.311576 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9mjp\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-kube-api-access-k9mjp\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.311618 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.311610 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.311618 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.311620 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357b85d9-1e44-4ab2-bdad-213169862e53-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.311618 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.311628 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.311913 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.311637 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357b85d9-1e44-4ab2-bdad-213169862e53-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.311913 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.311645 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357b85d9-1e44-4ab2-bdad-213169862e53-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.311913 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.311653 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357b85d9-1e44-4ab2-bdad-213169862e53-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:00:59.472731 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.472692 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd"] Jul 28 18:00:59.478432 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.478404 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-20ce8c-openshift-default-589cdd8664-288dd"] Jul 28 18:00:59.684939 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:00:59.684906 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" path="/var/lib/kubelet/pods/357b85d9-1e44-4ab2-bdad-213169862e53/volumes" Jul 28 18:01:05.714628 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.714588 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf"] Jul 28 18:01:05.715190 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.714853 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" podUID="adf8ada0-63db-4c20-a3d8-5c81c3af47b1" containerName="maas-api" containerID="cri-o://a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655" gracePeriod=30 Jul 28 18:01:05.746692 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.746663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:05.747333 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.747311 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd"] Jul 28 18:01:05.760873 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.760846 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-857ec3-29754360-4n8nd"] Jul 28 18:01:05.843328 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.842641 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:05.920680 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.920646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:05.978240 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:05.978162 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 18:01:06.073674 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.073635 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-openshift-service-ca\") pod \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " Jul 28 18:01:06.073674 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.073680 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-maas-api-tls\") pod \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " Jul 28 18:01:06.073946 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.073766 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-kube-api-access-xdn2q\") pod \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\" (UID: \"adf8ada0-63db-4c20-a3d8-5c81c3af47b1\") " Jul 28 18:01:06.074136 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.074080 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "adf8ada0-63db-4c20-a3d8-5c81c3af47b1" (UID: "adf8ada0-63db-4c20-a3d8-5c81c3af47b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:06.075879 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.075848 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "adf8ada0-63db-4c20-a3d8-5c81c3af47b1" (UID: "adf8ada0-63db-4c20-a3d8-5c81c3af47b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:01:06.075879 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.075847 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-kube-api-access-xdn2q" (OuterVolumeSpecName: "kube-api-access-xdn2q") pod "adf8ada0-63db-4c20-a3d8-5c81c3af47b1" (UID: "adf8ada0-63db-4c20-a3d8-5c81c3af47b1"). InnerVolumeSpecName "kube-api-access-xdn2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:01:06.174638 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.174594 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-kube-api-access-xdn2q\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:06.174638 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.174630 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-openshift-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:06.174638 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.174641 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf8ada0-63db-4c20-a3d8-5c81c3af47b1-maas-api-tls\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:06.185175 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.185141 2579 generic.go:356] "Generic (PLEG): container finished" podID="adf8ada0-63db-4c20-a3d8-5c81c3af47b1" containerID="a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655" exitCode=0 Jul 28 18:01:06.185349 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.185191 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" Jul 28 18:01:06.185349 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.185199 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" event={"ID":"adf8ada0-63db-4c20-a3d8-5c81c3af47b1","Type":"ContainerDied","Data":"a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655"} Jul 28 18:01:06.185349 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.185242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf" event={"ID":"adf8ada0-63db-4c20-a3d8-5c81c3af47b1","Type":"ContainerDied","Data":"aa9bae6019076a00f3f372e3e9383eaba8b37f691cf4f1d953c68dbc85eec1cd"} Jul 28 18:01:06.185349 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.185264 2579 scope.go:117] "RemoveContainer" containerID="a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655" Jul 28 18:01:06.194297 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.194277 2579 scope.go:117] "RemoveContainer" containerID="a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655" Jul 28 18:01:06.194604 ip-10-0-135-73 kubenswrapper[2579]: E0728 18:01:06.194585 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655\": container with ID starting with a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655 not found: ID does not exist" containerID="a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655" Jul 28 18:01:06.194656 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.194612 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655"} err="failed to get container status \"a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655\": rpc error: code = NotFound desc = could not find container \"a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655\": container with ID starting with a9816bf484c236a88d57c54b613c29d305969513ac114125e897ca4b025e6655 not found: ID does not exist" Jul 28 18:01:06.210126 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.210077 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf"] Jul 28 18:01:06.217928 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:06.217895 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-857ec3-5f95bb8865-hpzhf"] Jul 28 18:01:07.685847 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:07.685810 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="500efae2-0aec-4877-ba56-d52eb8341e16" path="/var/lib/kubelet/pods/500efae2-0aec-4877-ba56-d52eb8341e16/volumes" Jul 28 18:01:07.686263 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:07.686195 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adf8ada0-63db-4c20-a3d8-5c81c3af47b1" path="/var/lib/kubelet/pods/adf8ada0-63db-4c20-a3d8-5c81c3af47b1/volumes" Jul 28 18:01:08.948204 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:08.948164 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 18:01:09.421801 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:09.421760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:09.431626 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:09.431589 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf"] Jul 28 18:01:09.440028 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:09.439998 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-857ec3-889kf"] Jul 28 18:01:09.684599 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:09.684523 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f8d924-cdd2-44eb-9b13-fd2ead803d49" path="/var/lib/kubelet/pods/12f8d924-cdd2-44eb-9b13-fd2ead803d49/volumes" Jul 28 18:01:09.855777 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:09.855740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:17.567474 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:17.567441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:17.668301 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:17.668269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:17.872278 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:17.872237 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:18.106992 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.106958 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:18.271614 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.271511 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz"] Jul 28 18:01:18.271927 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.271909 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf8ada0-63db-4c20-a3d8-5c81c3af47b1" containerName="maas-api" Jul 28 18:01:18.272037 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.271929 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf8ada0-63db-4c20-a3d8-5c81c3af47b1" containerName="maas-api" Jul 28 18:01:18.272037 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.271960 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12f8d924-cdd2-44eb-9b13-fd2ead803d49" containerName="revoke-keys" Jul 28 18:01:18.272037 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.271970 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12f8d924-cdd2-44eb-9b13-fd2ead803d49" containerName="revoke-keys" Jul 28 18:01:18.272037 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.271992 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" Jul 28 18:01:18.272037 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.272000 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" Jul 28 18:01:18.272037 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.272013 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" Jul 28 18:01:18.272037 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.272022 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" Jul 28 18:01:18.272417 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.272102 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" Jul 28 18:01:18.272417 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.272145 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf8ada0-63db-4c20-a3d8-5c81c3af47b1" containerName="maas-api" Jul 28 18:01:18.272417 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.272157 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12f8d924-cdd2-44eb-9b13-fd2ead803d49" containerName="revoke-keys" Jul 28 18:01:18.276317 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.276289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" Jul 28 18:01:18.281001 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.280974 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:01:18.281001 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.280984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 18:01:18.294430 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.294391 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz"] Jul 28 18:01:18.374714 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.374668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz\" (UID: \"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" Jul 28 18:01:18.475151 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.475089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz\" (UID: \"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" Jul 28 18:01:18.475751 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.475729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz\" (UID: \"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" Jul 28 18:01:18.588160 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.588100 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" Jul 28 18:01:18.731075 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:18.731044 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz"] Jul 28 18:01:18.732998 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:01:18.732974 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44f69ca_5e88_4225_92ea_fcf0f1b6bcc9.slice/crio-06155b3a2b444dc1576be9108cf59dbe03c45d6888b0f00ae0987b17886b1c8d WatchSource:0}: Error finding container 06155b3a2b444dc1576be9108cf59dbe03c45d6888b0f00ae0987b17886b1c8d: Status 404 returned error can't find the container with id 06155b3a2b444dc1576be9108cf59dbe03c45d6888b0f00ae0987b17886b1c8d Jul 28 18:01:19.259595 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:19.259489 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" event={"ID":"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9","Type":"ContainerStarted","Data":"0618a1c2ae7634cdf879c478eaabccd16b0ddb3f8c56b59f4c87e9c0494ff658"} Jul 28 18:01:19.259595 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:19.259546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" event={"ID":"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9","Type":"ContainerStarted","Data":"06155b3a2b444dc1576be9108cf59dbe03c45d6888b0f00ae0987b17886b1c8d"} Jul 28 18:01:19.287159 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:19.287076 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" podStartSLOduration=1.287056658 podStartE2EDuration="1.287056658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:19.286752579 +0000 UTC m=+3176.090441169" watchObservedRunningTime="2026-07-28 18:01:19.287056658 +0000 UTC m=+3176.090745251" Jul 28 18:01:20.266246 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:20.266207 2579 generic.go:356] "Generic (PLEG): container finished" podID="d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9" containerID="0618a1c2ae7634cdf879c478eaabccd16b0ddb3f8c56b59f4c87e9c0494ff658" exitCode=0 Jul 28 18:01:20.266684 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:20.266294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" event={"ID":"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9","Type":"ContainerDied","Data":"0618a1c2ae7634cdf879c478eaabccd16b0ddb3f8c56b59f4c87e9c0494ff658"} Jul 28 18:01:21.402288 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:21.402261 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" Jul 28 18:01:21.499156 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:21.499123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9-maas-api-service-ca\") pod \"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9\" (UID: \"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9\") " Jul 28 18:01:21.499527 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:21.499503 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9" (UID: "d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:21.600235 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:21.600189 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:22.277729 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:22.277691 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" event={"ID":"d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9","Type":"ContainerDied","Data":"06155b3a2b444dc1576be9108cf59dbe03c45d6888b0f00ae0987b17886b1c8d"} Jul 28 18:01:22.277729 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:22.277725 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz" Jul 28 18:01:22.277970 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:22.277743 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06155b3a2b444dc1576be9108cf59dbe03c45d6888b0f00ae0987b17886b1c8d" Jul 28 18:01:29.251465 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:29.251425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:29.266732 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:29.266698 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg"] Jul 28 18:01:29.269005 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:29.268974 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-f0230c-29754360-fvmgg"] Jul 28 18:01:29.390303 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:29.390271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:29.498394 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:29.498360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:29.684604 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:29.684574 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f607985-19eb-46e0-829a-d203b8545a08" path="/var/lib/kubelet/pods/8f607985-19eb-46e0-829a-d203b8545a08/volumes" Jul 28 18:01:33.017870 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:33.017820 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 18:01:33.441282 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:33.441247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:33.477735 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:33.477692 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz"] Jul 28 18:01:33.482943 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:33.482914 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-f0230c-86lwz"] Jul 28 18:01:33.684596 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:33.684563 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9" path="/var/lib/kubelet/pods/d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9/volumes" Jul 28 18:01:34.035793 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:34.035758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:42.434057 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:42.434021 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:42.528945 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:42.528907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:42.686545 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:42.686443 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62"] Jul 28 18:01:42.686735 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:42.686706 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" containerID="cri-o://c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423" gracePeriod=30 Jul 28 18:01:42.704319 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:42.704283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:43.101295 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.101257 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww"] Jul 28 18:01:43.101576 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.101565 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9" containerName="revoke-keys" Jul 28 18:01:43.101626 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.101578 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9" containerName="revoke-keys" Jul 28 18:01:43.101662 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.101631 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44f69ca-5e88-4225-92ea-fcf0f1b6bcc9" containerName="revoke-keys" Jul 28 18:01:43.101662 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.101640 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="357b85d9-1e44-4ab2-bdad-213169862e53" containerName="istio-proxy" Jul 28 18:01:43.104597 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.104577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" Jul 28 18:01:43.107401 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.107377 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:01:43.107543 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.107402 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tl4gw\"" Jul 28 18:01:43.122715 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.122680 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww"] Jul 28 18:01:43.194405 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.194368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782fbc95-098d-448a-aded-80955b71bacf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww\" (UID: \"782fbc95-098d-448a-aded-80955b71bacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" Jul 28 18:01:43.295010 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.294968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782fbc95-098d-448a-aded-80955b71bacf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww\" (UID: \"782fbc95-098d-448a-aded-80955b71bacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" Jul 28 18:01:43.295643 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.295623 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782fbc95-098d-448a-aded-80955b71bacf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww\" (UID: \"782fbc95-098d-448a-aded-80955b71bacf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" Jul 28 18:01:43.415091 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.414988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" Jul 28 18:01:43.570576 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.570514 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww"] Jul 28 18:01:43.574020 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:01:43.573981 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782fbc95_098d_448a_aded_80955b71bacf.slice/crio-f56ab2cdff02cf1699cf604d7d9fb2f21a336dd2ebceab866975ba06667b1b33 WatchSource:0}: Error finding container f56ab2cdff02cf1699cf604d7d9fb2f21a336dd2ebceab866975ba06667b1b33: Status 404 returned error can't find the container with id f56ab2cdff02cf1699cf604d7d9fb2f21a336dd2ebceab866975ba06667b1b33 Jul 28 18:01:43.852235 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:43.852204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:44.399347 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:44.399316 2579 generic.go:356] "Generic (PLEG): container finished" podID="782fbc95-098d-448a-aded-80955b71bacf" containerID="ec2012f172d9d1a7207fcdee4132269912a6fe84a303c51cab5051dd512369c6" exitCode=0 Jul 28 18:01:44.399533 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:44.399353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" event={"ID":"782fbc95-098d-448a-aded-80955b71bacf","Type":"ContainerDied","Data":"ec2012f172d9d1a7207fcdee4132269912a6fe84a303c51cab5051dd512369c6"} Jul 28 18:01:44.399533 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:44.399392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" event={"ID":"782fbc95-098d-448a-aded-80955b71bacf","Type":"ContainerStarted","Data":"f56ab2cdff02cf1699cf604d7d9fb2f21a336dd2ebceab866975ba06667b1b33"} Jul 28 18:01:45.537786 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:45.537755 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" Jul 28 18:01:45.615368 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:45.615330 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782fbc95-098d-448a-aded-80955b71bacf-maas-api-service-ca\") pod \"782fbc95-098d-448a-aded-80955b71bacf\" (UID: \"782fbc95-098d-448a-aded-80955b71bacf\") " Jul 28 18:01:45.615706 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:45.615682 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782fbc95-098d-448a-aded-80955b71bacf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "782fbc95-098d-448a-aded-80955b71bacf" (UID: "782fbc95-098d-448a-aded-80955b71bacf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:45.716427 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:45.716386 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/782fbc95-098d-448a-aded-80955b71bacf-maas-api-service-ca\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:46.411159 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:46.411127 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" Jul 28 18:01:46.411346 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:46.411129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww" event={"ID":"782fbc95-098d-448a-aded-80955b71bacf","Type":"ContainerDied","Data":"f56ab2cdff02cf1699cf604d7d9fb2f21a336dd2ebceab866975ba06667b1b33"} Jul 28 18:01:46.411346 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:46.411254 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f56ab2cdff02cf1699cf604d7d9fb2f21a336dd2ebceab866975ba06667b1b33" Jul 28 18:01:47.946014 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:47.945987 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 18:01:48.038596 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038507 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istiod-ca-cert\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038596 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038547 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-data\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038596 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038593 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-envoy\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038890 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038620 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf4bq\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-kube-api-access-wf4bq\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038890 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038651 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-certs\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038890 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038666 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-socket\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038890 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038691 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-podinfo\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038890 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038730 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-token\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.038890 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038749 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-credential-socket\") pod \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\" (UID: \"cc7dd300-afa6-4e3d-b97a-457165cc1fc2\") " Jul 28 18:01:48.039250 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.038961 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:48.039250 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.039182 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:01:48.039250 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.039227 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:01:48.039430 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.039298 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:01:48.039722 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.039696 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-data" (OuterVolumeSpecName: "istio-data") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:01:48.041008 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.040975 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:01:48.041008 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.040983 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:01:48.041177 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.041024 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:01:48.041177 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.041084 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-kube-api-access-wf4bq" (OuterVolumeSpecName: "kube-api-access-wf4bq") pod "cc7dd300-afa6-4e3d-b97a-457165cc1fc2" (UID: "cc7dd300-afa6-4e3d-b97a-457165cc1fc2"). InnerVolumeSpecName "kube-api-access-wf4bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:01:48.140403 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140363 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istiod-ca-cert\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140403 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140396 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-data\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140403 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140406 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-envoy\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140403 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140414 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf4bq\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-kube-api-access-wf4bq\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140695 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140424 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-certs\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140695 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140433 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-workload-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140695 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140442 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-podinfo\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140695 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140450 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-istio-token\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.140695 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.140457 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc7dd300-afa6-4e3d-b97a-457165cc1fc2-credential-socket\") on node \"ip-10-0-135-73.ec2.internal\" DevicePath \"\"" Jul 28 18:01:48.423276 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.423239 2579 generic.go:356] "Generic (PLEG): container finished" podID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerID="c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423" exitCode=0 Jul 28 18:01:48.423461 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.423289 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" Jul 28 18:01:48.423461 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.423318 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" event={"ID":"cc7dd300-afa6-4e3d-b97a-457165cc1fc2","Type":"ContainerDied","Data":"c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423"} Jul 28 18:01:48.423461 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.423374 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62" event={"ID":"cc7dd300-afa6-4e3d-b97a-457165cc1fc2","Type":"ContainerDied","Data":"e7a5abbe903b4a2363a123adfbf590b611e3b01796dc18c470fb9129213f6f42"} Jul 28 18:01:48.423461 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.423397 2579 scope.go:117] "RemoveContainer" containerID="c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423" Jul 28 18:01:48.434904 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.434873 2579 scope.go:117] "RemoveContainer" containerID="2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6" Jul 28 18:01:48.443903 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.443882 2579 scope.go:117] "RemoveContainer" containerID="c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423" Jul 28 18:01:48.444205 ip-10-0-135-73 kubenswrapper[2579]: E0728 18:01:48.444184 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423\": container with ID starting with c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423 not found: ID does not exist" containerID="c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423" Jul 28 18:01:48.444283 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.444215 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423"} err="failed to get container status \"c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423\": rpc error: code = NotFound desc = could not find container \"c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423\": container with ID starting with c70f63fb170da7d8dcb40a77c7445d65185cc94c006919799ce77c56e7287423 not found: ID does not exist" Jul 28 18:01:48.444283 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.444235 2579 scope.go:117] "RemoveContainer" containerID="2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6" Jul 28 18:01:48.444467 ip-10-0-135-73 kubenswrapper[2579]: E0728 18:01:48.444446 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6\": container with ID starting with 2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6 not found: ID does not exist" containerID="2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6" Jul 28 18:01:48.444518 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.444473 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6"} err="failed to get container status \"2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6\": rpc error: code = NotFound desc = could not find container \"2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6\": container with ID starting with 2b8233cd20676c6845140ea305aace6a54355656e50e93326ec31ec13ec1c5f6 not found: ID does not exist" Jul 28 18:01:48.451908 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.451885 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62"] Jul 28 18:01:48.453794 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:48.453774 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-f0230c-openshift-default-74d5575bdd-6bq62"] Jul 28 18:01:49.684890 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:49.684852 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" path="/var/lib/kubelet/pods/cc7dd300-afa6-4e3d-b97a-457165cc1fc2/volumes" Jul 28 18:01:53.250864 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:53.250825 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h"] Jul 28 18:01:53.252621 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:53.252598 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-f24b7c-29754360-rk25h"] Jul 28 18:01:53.684365 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:53.684330 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56bcbed7-d878-46df-9a9f-e50b51514fb7" path="/var/lib/kubelet/pods/56bcbed7-d878-46df-9a9f-e50b51514fb7/volumes" Jul 28 18:01:54.343261 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:54.343227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:54.419132 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:54.419069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:56.107209 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:56.107176 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-khkq9"] Jul 28 18:01:58.196503 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:58.196467 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:58.196955 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:58.196672 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww"] Jul 28 18:01:58.199732 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:58.199699 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f24b7c-85lww"] Jul 28 18:01:58.599231 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:58.599181 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:59.684212 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:01:59.684181 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="782fbc95-098d-448a-aded-80955b71bacf" path="/var/lib/kubelet/pods/782fbc95-098d-448a-aded-80955b71bacf/volumes" Jul 28 18:02:05.665036 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:05.664985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:06.775153 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:06.775101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:13.337601 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:13.337568 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-q26l5_a50cf5e2-75e3-4927-8b16-b49bd2d57b59/manager/0.log" Jul 28 18:02:13.354149 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:13.354122 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-q26l5_a50cf5e2-75e3-4927-8b16-b49bd2d57b59/copy-manifests/0.log" Jul 28 18:02:13.832318 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:13.832226 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754330-zjx8l_0d046c02-d6a3-41a7-8647-4083d1dc5f82/cleanup/0.log" Jul 28 18:02:14.071991 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:14.071948 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754360-zhn6p_0449f0d7-5d54-4d2a-9621-547cd1cc767f/cleanup/0.log" Jul 28 18:02:15.000770 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.000737 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp_6a8fecb2-2a21-40ef-aa06-9b4c84535295/pull/0.log" Jul 28 18:02:15.009402 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.009375 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp_6a8fecb2-2a21-40ef-aa06-9b4c84535295/extract/0.log" Jul 28 18:02:15.017462 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.017431 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp_6a8fecb2-2a21-40ef-aa06-9b4c84535295/util/0.log" Jul 28 18:02:15.140558 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.140526 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8_7c8ac905-db82-43da-812c-9bbac86ee418/pull/0.log" Jul 28 18:02:15.148983 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.148959 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8_7c8ac905-db82-43da-812c-9bbac86ee418/extract/0.log" Jul 28 18:02:15.157944 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.157921 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8_7c8ac905-db82-43da-812c-9bbac86ee418/util/0.log" Jul 28 18:02:15.273266 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.273184 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd_fc612252-cc0f-4016-9b05-33cbf44cfc97/util/0.log" Jul 28 18:02:15.281347 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.281327 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd_fc612252-cc0f-4016-9b05-33cbf44cfc97/pull/0.log" Jul 28 18:02:15.292099 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.292065 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd_fc612252-cc0f-4016-9b05-33cbf44cfc97/extract/0.log" Jul 28 18:02:15.419669 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.419640 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg_e62a862c-3584-4f5c-a084-0a29427876d7/util/0.log" Jul 28 18:02:15.428030 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.428007 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg_e62a862c-3584-4f5c-a084-0a29427876d7/pull/0.log" Jul 28 18:02:15.436535 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:15.436502 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg_e62a862c-3584-4f5c-a084-0a29427876d7/extract/0.log" Jul 28 18:02:16.297349 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:16.297317 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-khkq9_963ca571-246f-42fb-af69-a05ad6dde577/limitador/0.log" Jul 28 18:02:16.947714 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:16.947675 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lqs6m_0185fa99-d0d2-44c2-8747-db80d71a5ab2/discovery/0.log" Jul 28 18:02:18.081216 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:18.081176 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-t7gs5_1d6d6f46-4390-4278-8b38-5e6b2c7dd7ab/main/0.log" Jul 28 18:02:24.372131 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:24.372004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:24.449456 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:24.449422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:27.328244 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:27.328203 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-djz6z_5f6c9d40-ea89-4929-9a71-eb0ca0a4ff42/global-pull-secret-syncer/0.log" Jul 28 18:02:27.484624 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:27.484586 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" Jul 28 18:02:27.645741 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:27.645703 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-73.ec2.internal_df396ec83974a4b79253cc8ed0dbbedd/haproxy/0.log" Jul 28 18:02:31.963292 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:31.963260 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp_6a8fecb2-2a21-40ef-aa06-9b4c84535295/extract/0.log" Jul 28 18:02:31.998321 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:31.998288 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp_6a8fecb2-2a21-40ef-aa06-9b4c84535295/util/0.log" Jul 28 18:02:32.043179 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.043148 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q4ntp_6a8fecb2-2a21-40ef-aa06-9b4c84535295/pull/0.log" Jul 28 18:02:32.084364 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.084331 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8_7c8ac905-db82-43da-812c-9bbac86ee418/extract/0.log" Jul 28 18:02:32.117283 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.117254 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8_7c8ac905-db82-43da-812c-9bbac86ee418/util/0.log" Jul 28 18:02:32.155427 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.155395 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0lxpx8_7c8ac905-db82-43da-812c-9bbac86ee418/pull/0.log" Jul 28 18:02:32.193076 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.193042 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd_fc612252-cc0f-4016-9b05-33cbf44cfc97/extract/0.log" Jul 28 18:02:32.221565 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.221469 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd_fc612252-cc0f-4016-9b05-33cbf44cfc97/util/0.log" Jul 28 18:02:32.249032 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.249004 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l9gnd_fc612252-cc0f-4016-9b05-33cbf44cfc97/pull/0.log" Jul 28 18:02:32.292239 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.292200 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg_e62a862c-3584-4f5c-a084-0a29427876d7/extract/0.log" Jul 28 18:02:32.325685 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.325657 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg_e62a862c-3584-4f5c-a084-0a29427876d7/util/0.log" Jul 28 18:02:32.361670 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:32.361631 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q6xdg_e62a862c-3584-4f5c-a084-0a29427876d7/pull/0.log" Jul 28 18:02:33.171103 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:33.171069 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-khkq9_963ca571-246f-42fb-af69-a05ad6dde577/limitador/0.log" Jul 28 18:02:35.543456 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:35.543422 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jst_6b394bff-b7cd-4f8c-a141-b5daefcc13e3/node-exporter/0.log" Jul 28 18:02:35.568418 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:35.568376 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jst_6b394bff-b7cd-4f8c-a141-b5daefcc13e3/kube-rbac-proxy/0.log" Jul 28 18:02:35.594040 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:35.594010 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8jst_6b394bff-b7cd-4f8c-a141-b5daefcc13e3/init-textfile/0.log" Jul 28 18:02:36.220080 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:36.220040 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b7bf6658c-d4rld_68ffba35-1a43-4953-8bc9-c8a99e766b06/telemeter-client/0.log" Jul 28 18:02:36.256332 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:36.256300 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b7bf6658c-d4rld_68ffba35-1a43-4953-8bc9-c8a99e766b06/reload/0.log" Jul 28 18:02:36.284714 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:36.284659 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b7bf6658c-d4rld_68ffba35-1a43-4953-8bc9-c8a99e766b06/kube-rbac-proxy/0.log" Jul 28 18:02:37.264673 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.264638 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5"] Jul 28 18:02:37.265264 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.265059 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" Jul 28 18:02:37.265264 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.265077 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" Jul 28 18:02:37.265264 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.265100 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="782fbc95-098d-448a-aded-80955b71bacf" containerName="revoke-keys" Jul 28 18:02:37.265264 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.265124 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="782fbc95-098d-448a-aded-80955b71bacf" containerName="revoke-keys" Jul 28 18:02:37.265264 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.265202 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" Jul 28 18:02:37.265264 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.265215 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7dd300-afa6-4e3d-b97a-457165cc1fc2" containerName="istio-proxy" Jul 28 18:02:37.265264 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.265227 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="782fbc95-098d-448a-aded-80955b71bacf" containerName="revoke-keys" Jul 28 18:02:37.267335 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.267318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.270240 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.270218 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f5qzw\"/\"openshift-service-ca.crt\"" Jul 28 18:02:37.271525 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.271509 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-f5qzw\"/\"default-dockercfg-rmzjs\"" Jul 28 18:02:37.271658 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.271546 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-f5qzw\"/\"kube-root-ca.crt\"" Jul 28 18:02:37.279468 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.279438 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5"] Jul 28 18:02:37.359089 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.359035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-sys\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.359089 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.359095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbtxm\" (UniqueName: \"kubernetes.io/projected/638efa51-f3c9-4492-8210-4a910ed7cda4-kube-api-access-nbtxm\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.359346 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.359191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-proc\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.359346 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.359229 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-podres\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.359346 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.359261 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-lib-modules\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460326 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-sys\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460326 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbtxm\" (UniqueName: \"kubernetes.io/projected/638efa51-f3c9-4492-8210-4a910ed7cda4-kube-api-access-nbtxm\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460533 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-proc\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460533 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-podres\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460533 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460421 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-sys\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460533 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-lib-modules\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460533 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460480 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-proc\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460729 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460552 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-podres\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.460729 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.460582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/638efa51-f3c9-4492-8210-4a910ed7cda4-lib-modules\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.469026 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.468991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbtxm\" (UniqueName: \"kubernetes.io/projected/638efa51-f3c9-4492-8210-4a910ed7cda4-kube-api-access-nbtxm\") pod \"perf-node-gather-daemonset-xjfs5\" (UID: \"638efa51-f3c9-4492-8210-4a910ed7cda4\") " pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.577938 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.577833 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:37.712787 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:37.712750 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5"] Jul 28 18:02:37.714595 ip-10-0-135-73 kubenswrapper[2579]: W0728 18:02:37.714553 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod638efa51_f3c9_4492_8210_4a910ed7cda4.slice/crio-a74e7d1e26b1a86bb3b92fc79cedda9bf5dcefa1d3db14a809d0c03bfb7757a9 WatchSource:0}: Error finding container a74e7d1e26b1a86bb3b92fc79cedda9bf5dcefa1d3db14a809d0c03bfb7757a9: Status 404 returned error can't find the container with id a74e7d1e26b1a86bb3b92fc79cedda9bf5dcefa1d3db14a809d0c03bfb7757a9 Jul 28 18:02:38.696365 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:38.696328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" event={"ID":"638efa51-f3c9-4492-8210-4a910ed7cda4","Type":"ContainerStarted","Data":"49c4d9c9715a4c274fd71e1e4bb48aab15109e5a13d75a9bd7db97e79c36ab11"} Jul 28 18:02:38.696792 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:38.696405 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" event={"ID":"638efa51-f3c9-4492-8210-4a910ed7cda4","Type":"ContainerStarted","Data":"a74e7d1e26b1a86bb3b92fc79cedda9bf5dcefa1d3db14a809d0c03bfb7757a9"} Jul 28 18:02:38.696792 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:38.696431 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:38.728005 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:38.727949 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" podStartSLOduration=1.727934531 podStartE2EDuration="1.727934531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:02:38.726381135 +0000 UTC m=+3255.530069725" watchObservedRunningTime="2026-07-28 18:02:38.727934531 +0000 UTC m=+3255.531623121" Jul 28 18:02:38.850233 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:38.850198 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-f99699b64-9khs8_a763eba7-3c81-41d9-b75f-febb5f85d8eb/console/0.log" Jul 28 18:02:38.903620 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:38.903574 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-kh899_f653f775-cac5-44a4-9d6b-aa2ed0866f34/download-server/0.log" Jul 28 18:02:40.898475 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:40.898445 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lv5jl_3c240724-e4a1-4888-a88c-099f5ecddf3d/dns/0.log" Jul 28 18:02:40.987944 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:40.987914 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lv5jl_3c240724-e4a1-4888-a88c-099f5ecddf3d/kube-rbac-proxy/0.log" Jul 28 18:02:41.140743 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:41.140713 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v9vdk_a3b34bff-2ba4-4bf3-8b96-fb67b5bc3c49/dns-node-resolver/0.log" Jul 28 18:02:41.778806 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:41.778774 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6clsf_207e36ae-5860-4aa8-841b-a3a8ce36da7d/node-ca/0.log" Jul 28 18:02:43.161385 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:43.161349 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lqs6m_0185fa99-d0d2-44c2-8747-db80d71a5ab2/discovery/0.log" Jul 28 18:02:44.457424 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:44.457391 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vb2ps_a19fa65d-1797-4cf1-bfd2-ad05664b46dc/serve-healthcheck-canary/0.log" Jul 28 18:02:44.656009 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:44.655976 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:44.710780 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:44.710695 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-f5qzw/perf-node-gather-daemonset-xjfs5" Jul 28 18:02:44.744030 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:44.743999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b585j_a727c9ba-bd47-42de-b962-dfb4560caf3c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:45.062515 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:45.062431 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9mhvt_9846a367-9fc2-4345-8d5c-2fde547331f4/kube-rbac-proxy/0.log" Jul 28 18:02:45.124672 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:45.124642 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9mhvt_9846a367-9fc2-4345-8d5c-2fde547331f4/exporter/0.log" Jul 28 18:02:45.181335 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:45.181306 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9mhvt_9846a367-9fc2-4345-8d5c-2fde547331f4/extractor/0.log" Jul 28 18:02:48.170873 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:48.170834 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lqs6m_0185fa99-d0d2-44c2-8747-db80d71a5ab2/discovery/0.log" Jul 28 18:02:49.162365 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:49.162327 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-q26l5_a50cf5e2-75e3-4927-8b16-b49bd2d57b59/manager/0.log" Jul 28 18:02:49.187967 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:49.187937 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d8c7b7dcd-q26l5_a50cf5e2-75e3-4927-8b16-b49bd2d57b59/copy-manifests/0.log" Jul 28 18:02:50.004225 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:50.004188 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-lqs6m_0185fa99-d0d2-44c2-8747-db80d71a5ab2/discovery/0.log" Jul 28 18:02:50.539785 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:50.539752 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5cdc965c4d-lf448_946fa23d-66f5-4ec5-b90e-eedb5a21f701/manager/0.log" Jul 28 18:02:50.627960 ip-10-0-135-73 kubenswrapper[2579]: I0728 18:02:50.627923 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-w46nk_21bdaf95-f0ad-4c98-9607-a570696f4c02/openshift-lws-operator/0.log"