Jul 15 19:14:24.640164 ip-10-0-132-25 systemd[1]: Starting Kubernetes Kubelet... Jul 15 19:14:24.952960 ip-10-0-132-25 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:14:24.952960 ip-10-0-132-25 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 19:14:24.952960 ip-10-0-132-25 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:14:24.952960 ip-10-0-132-25 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 19:14:24.952960 ip-10-0-132-25 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 19:14:24.954430 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.954316 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 19:14:24.959282 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959268 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:14:24.959282 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959282 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959286 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959289 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959292 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959295 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959298 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959301 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959303 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959306 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959309 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959312 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959315 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959317 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959320 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959328 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959330 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959333 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959336 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959338 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:14:24.959344 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959343 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959346 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959351 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959355 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959359 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959362 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959364 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959367 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959370 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959374 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959377 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959379 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959382 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959384 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959387 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959390 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959392 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959395 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959397 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959400 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:14:24.959799 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959402 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959405 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959408 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959410 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959412 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959415 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959418 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959420 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959423 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959425 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959428 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959430 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959433 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959436 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959439 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959442 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959445 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959448 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959450 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:14:24.960314 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959454 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959457 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959460 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959463 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959466 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959469 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959471 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959474 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959476 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959478 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959481 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959483 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959487 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959490 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959492 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959495 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959497 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959500 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959503 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959506 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:14:24.960813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959508 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959511 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959513 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959516 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959518 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959521 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959524 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959527 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959529 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959532 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959535 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959537 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959539 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959542 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959544 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959547 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959550 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959552 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959555 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:14:24.961318 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.959557 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960253 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960260 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960263 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960266 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960269 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960272 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960274 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960277 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960280 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960283 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960286 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960289 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960292 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960295 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960298 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960300 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960303 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960306 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960309 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:14:24.961790 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960312 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960314 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960317 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960320 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960322 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960325 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960329 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960332 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960335 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960338 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960340 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960343 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960346 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960348 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960350 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960353 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960357 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960361 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960364 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:14:24.962300 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960367 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960369 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960372 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960375 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960377 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960380 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960383 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960385 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960388 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960390 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960393 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960395 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960399 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960401 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960404 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960406 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960409 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960411 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960414 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960416 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:14:24.962754 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960419 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960421 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960424 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960426 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960429 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960431 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960434 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960436 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960439 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960441 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960444 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960447 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960450 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960452 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960455 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960457 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960460 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960462 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960466 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960469 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:14:24.963260 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960472 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960474 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960477 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960479 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960482 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960484 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960487 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960489 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960492 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960494 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960497 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960499 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960502 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960507 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960509 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960512 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960514 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960517 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960521 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960525 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:14:24.963743 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960528 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960894 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960914 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960918 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960921 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960923 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960926 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960929 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960932 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960935 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960937 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960940 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960943 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960946 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960949 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960952 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960955 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960957 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960960 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960963 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:14:24.964255 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960965 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960968 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960970 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960973 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960975 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960980 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960982 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960985 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960987 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960990 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960992 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.960997 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961000 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961003 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961006 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961009 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961011 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961014 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961016 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:14:24.964729 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961019 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961021 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961024 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961026 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961029 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961033 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961037 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961042 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961045 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961047 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961050 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961053 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961055 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961057 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961060 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961062 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961065 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961075 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961079 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:14:24.965211 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961082 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961084 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961087 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961089 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961093 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961095 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961097 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961100 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961102 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961105 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961107 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961111 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961114 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961117 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961120 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961122 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961125 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961127 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961130 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:14:24.965667 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961136 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961141 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961144 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961147 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961149 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961152 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961154 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961157 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961160 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961162 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961165 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961167 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961171 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961173 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961175 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961178 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961180 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961183 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961186 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961188 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:14:24.966135 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961190 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961193 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961195 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961270 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961280 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961290 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961296 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961302 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961306 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961313 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961318 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961322 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961325 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961330 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961333 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961337 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961340 2568 flags.go:64] FLAG: --cgroup-root="" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961343 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961346 2568 flags.go:64] FLAG: --client-ca-file="" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961349 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961352 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961355 2568 flags.go:64] FLAG: --cluster-domain="" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961358 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961361 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 19:14:24.966632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961365 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961369 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961372 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961375 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961378 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961382 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961385 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961388 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961390 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961394 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961397 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961400 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961403 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961406 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961409 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961412 2568 flags.go:64] FLAG: --enable-server="true" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961415 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961419 2568 flags.go:64] FLAG: --event-burst="100" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961422 2568 flags.go:64] FLAG: --event-qps="50" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961425 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961428 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961431 2568 flags.go:64] FLAG: --eviction-hard="" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961435 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961438 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961441 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 19:14:24.967249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961444 2568 flags.go:64] FLAG: --eviction-soft="" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961447 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961450 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961453 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961456 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961459 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961462 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961467 2568 flags.go:64] FLAG: --feature-gates="" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961470 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961474 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961477 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961481 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961484 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961487 2568 flags.go:64] FLAG: --help="false" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961490 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-132-25.ec2.internal" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961493 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961496 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961499 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961502 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961505 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961508 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961511 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961514 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 19:14:24.967833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961517 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961519 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961522 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961525 2568 flags.go:64] FLAG: --kube-reserved="" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961528 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961531 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961534 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961538 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961541 2568 flags.go:64] FLAG: --lock-file="" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961544 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961547 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961550 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961555 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961557 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961560 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961563 2568 flags.go:64] FLAG: --logging-format="text" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961567 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961571 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961573 2568 flags.go:64] FLAG: --manifest-url="" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961576 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961581 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961584 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961588 2568 flags.go:64] FLAG: --max-pods="110" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961591 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961594 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 19:14:24.968417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961597 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961600 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961603 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961606 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961609 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961616 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961619 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961622 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961625 2568 flags.go:64] FLAG: --pod-cidr="" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961628 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961633 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961636 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961639 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961642 2568 flags.go:64] FLAG: --port="10250" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961645 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961649 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07acab0d036df870a" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961652 2568 flags.go:64] FLAG: --qos-reserved="" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961655 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961658 2568 flags.go:64] FLAG: --register-node="true" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961661 2568 flags.go:64] FLAG: --register-with-taints="" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961664 2568 flags.go:64] FLAG: --registry-burst="10" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961668 2568 flags.go:64] FLAG: --registry-qps="5" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961670 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961676 2568 flags.go:64] FLAG: --reserved-memory="" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961679 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 19:14:24.969141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961682 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961685 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961688 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961691 2568 flags.go:64] FLAG: --runonce="false" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961694 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961697 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961700 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961703 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961706 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961709 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961712 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961715 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961718 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961721 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961724 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961726 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961730 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961733 2568 flags.go:64] FLAG: --system-cgroups="" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961735 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961742 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961745 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961748 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961751 2568 flags.go:64] FLAG: --tls-min-version="" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961755 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961757 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 19:14:24.969742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961761 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961763 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961766 2568 flags.go:64] FLAG: --v="2" Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961770 2568 flags.go:64] FLAG: --version="false" Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961774 2568 flags.go:64] FLAG: --vmodule="" Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961779 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.961783 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961924 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961928 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961932 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961935 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961938 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961941 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961944 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961947 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961952 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961955 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961958 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961961 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961963 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:14:24.970362 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961966 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961968 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961971 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961974 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961976 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961979 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961983 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961986 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961989 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961991 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961994 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.961997 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962000 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962002 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962005 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962007 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962010 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962013 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962016 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:14:24.970852 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962019 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962021 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962024 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962026 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962029 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962031 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962034 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962036 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962039 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962041 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962044 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962046 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962049 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962051 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962054 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962056 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962059 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962061 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962063 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962067 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:14:24.971339 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962070 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962073 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962075 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962078 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962080 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962085 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962088 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962091 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962094 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962097 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962100 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962103 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962106 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962108 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962111 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962113 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962116 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962118 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962121 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962123 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:14:24.971870 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962125 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962128 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962130 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962133 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962136 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962139 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962141 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962143 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962146 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962149 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962151 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962155 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962158 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962160 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962163 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962165 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962168 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962170 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962173 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962176 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:14:24.972370 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962178 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962181 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962183 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962186 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962189 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962192 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.962196 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.962200 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.968275 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.968289 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.968920 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.968927 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.968982 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.968988 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:14:24.972897 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.968992 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.968995 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.968998 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969002 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969005 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969008 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969010 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969013 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969015 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969018 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969021 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969024 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969026 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969029 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969031 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969034 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969036 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969039 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969042 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:14:24.973277 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969044 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969047 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969049 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969052 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969055 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969057 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969060 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969063 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969073 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969077 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969080 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969082 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969085 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969087 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969090 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969093 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969096 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969098 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969101 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969103 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:14:24.973740 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969106 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969109 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969111 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969114 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969116 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969119 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969121 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969124 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969126 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969129 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969131 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969134 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969136 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969139 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969143 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969147 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969150 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969153 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969156 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:14:24.974346 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969159 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969163 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969168 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969171 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969174 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969176 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969179 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969181 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969184 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969187 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969190 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969192 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969195 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969198 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969200 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969203 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969205 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969208 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969210 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969213 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:14:24.974813 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969216 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969218 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969221 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969224 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969226 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969229 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969232 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969234 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969237 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969240 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969242 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969246 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969249 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969252 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969255 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969258 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969261 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969264 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:14:24.975316 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969266 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.969271 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969379 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969384 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969387 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969390 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969394 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969397 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969400 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969402 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969405 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969407 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969410 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969413 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 19:14:24.975756 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969415 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969417 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969420 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969422 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969425 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969427 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969430 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969432 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969435 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969437 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969440 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969442 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969445 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969447 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969451 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969454 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969458 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969461 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969464 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 19:14:24.976222 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969466 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969469 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969471 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969474 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969476 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969479 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969481 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969483 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969486 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969488 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969491 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969493 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969496 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969498 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969500 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969503 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969506 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969508 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969511 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969513 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 19:14:24.976990 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969516 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969518 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969521 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969523 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969527 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969531 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969534 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969536 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969539 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969542 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969548 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969550 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969553 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969555 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969558 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969560 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969563 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969565 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969568 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 19:14:24.977486 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969570 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969573 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969575 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969578 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969581 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969583 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969586 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969589 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969591 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969594 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969596 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969598 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969601 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969604 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969606 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969608 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969611 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969613 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969619 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969621 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 19:14:24.977953 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969624 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969626 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969629 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969631 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969635 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969637 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969640 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969642 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:24.969645 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.969650 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.969745 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.971598 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.972606 2568 server.go:1019] "Starting client certificate rotation" Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.972706 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:14:24.978468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.973288 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 19:14:24.978807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.974884 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:14:24.979934 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.979916 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 19:14:24.988183 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.988165 2568 log.go:25] "Validated CRI v1 runtime API" Jul 15 19:14:24.990023 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.990004 2568 log.go:25] "Validated CRI v1 image API" Jul 15 19:14:24.990781 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.990767 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 19:14:24.994755 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.994735 2568 fs.go:135] Filesystem UUIDs: map[6f16b6df-299e-4700-b472-bb62fcfb9c31:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8e182b86-f339-4567-ada7-b02538ab5067:/dev/nvme0n1p4] Jul 15 19:14:24.994831 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.994755 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 19:14:24.998860 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.998747 2568 manager.go:216] Machine: {Timestamp:2026-07-15 19:14:24.998297213 +0000 UTC m=+0.266616857 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103733 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b3bbddb75d7cb1198aabc6a715415 SystemUUID:ec2b3bbd-db75-d7cb-1198-aabc6a715415 BootID:dc474ac2-2c5a-424d-a606-abcaa97dcc5f Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:5d:3c:c3:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:5d:3c:c3:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:25:42:48:2a:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 19:14:24.998860 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.998847 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 19:14:24.999025 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:24.998951 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 19:14:25.001645 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.001624 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 19:14:25.001777 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.001648 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-25.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 19:14:25.002195 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.002186 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 19:14:25.002228 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.002198 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 19:14:25.002228 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.002205 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:14:25.002228 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.002217 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 19:14:25.003319 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.003309 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 19:14:25.007046 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007035 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:14:25.007147 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007138 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 19:14:25.007176 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007163 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 15 19:14:25.007531 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007521 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 19:14:25.007562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007521 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:14:25.007562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007544 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 19:14:25.007562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007553 2568 kubelet.go:398] "Adding apiserver pod source" Jul 15 19:14:25.007562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007561 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 19:14:25.007999 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.007976 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:14:25.008094 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.008065 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 19:14:25.010353 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.010327 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 19:14:25.012102 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012081 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 19:14:25.012152 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012118 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 19:14:25.012276 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012267 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 19:14:25.012321 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012281 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 19:14:25.012321 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012287 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 19:14:25.012321 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012293 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 19:14:25.012321 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012301 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 19:14:25.012321 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012307 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 19:14:25.012718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012708 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 19:14:25.012718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012718 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 19:14:25.012813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012729 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 19:14:25.012813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012735 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 19:14:25.012813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012744 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 19:14:25.012813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.012752 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 19:14:25.014142 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.014126 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 19:14:25.014213 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.014161 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 19:14:25.017772 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.017755 2568 server.go:1268] "Started kubelet" Jul 15 19:14:25.017868 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.017810 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 19:14:25.017972 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.017952 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 19:14:25.018020 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.017974 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 19:14:25.018701 ip-10-0-132-25 systemd[1]: Started Kubernetes Kubelet. Jul 15 19:14:25.019080 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.019061 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 19:14:25.020466 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.020410 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 19:14:25.020466 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.020441 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 19:14:25.020612 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.020491 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-25.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 19:14:25.021218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.021200 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 15 19:14:25.022706 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.022684 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 19:14:25.025108 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.025091 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 19:14:25.025184 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.025093 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 19:14:25.025235 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.025173 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:25.025591 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.025571 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 19:14:25.025681 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.025601 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 19:14:25.025681 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.025626 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 19:14:25.025774 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.025747 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 15 19:14:25.025774 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.025759 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 15 19:14:25.026249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026230 2568 factory.go:55] Registering systemd factory Jul 15 19:14:25.026249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026252 2568 factory.go:223] Registration of the systemd container factory successfully Jul 15 19:14:25.026462 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026447 2568 factory.go:153] Registering CRI-O factory Jul 15 19:14:25.026526 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026466 2568 factory.go:223] Registration of the crio container factory successfully Jul 15 19:14:25.026526 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026510 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 19:14:25.026622 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026532 2568 factory.go:103] Registering Raw factory Jul 15 19:14:25.026622 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026551 2568 manager.go:1196] Started watching for new ooms in manager Jul 15 19:14:25.026919 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.026893 2568 manager.go:349] Starting recovery of all containers Jul 15 19:14:25.027018 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.026130 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-25.ec2.internal.18c28c7ba1edff63 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-25.ec2.internal,UID:ip-10-0-132-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-25.ec2.internal,},FirstTimestamp:2026-07-15 19:14:25.017732963 +0000 UTC m=+0.286052588,LastTimestamp:2026-07-15 19:14:25.017732963 +0000 UTC m=+0.286052588,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-25.ec2.internal,}" Jul 15 19:14:25.031646 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.031613 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-25.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 19:14:25.031742 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.031723 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 19:14:25.032119 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.032094 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fcqhn" Jul 15 19:14:25.036312 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.036295 2568 manager.go:354] Recovery completed Jul 15 19:14:25.040771 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.040753 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fcqhn" Jul 15 19:14:25.042325 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.042312 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:14:25.044097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.044048 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:14:25.044097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.044069 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:14:25.044097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.044084 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:14:25.044645 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.044628 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 19:14:25.044645 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.044645 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 19:14:25.044789 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.044661 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 15 19:14:25.044789 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.044632 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:14:25.045493 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.045477 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:14:25.045574 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.045506 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:14:25.045574 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.045521 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:14:25.046442 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:25.046382 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-25.ec2.internal.18c28c7ba37fb763 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-25.ec2.internal,UID:ip-10-0-132-25.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-25.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-25.ec2.internal,},FirstTimestamp:2026-07-15 19:14:25.044060003 +0000 UTC m=+0.312379626,LastTimestamp:2026-07-15 19:14:25.044060003 +0000 UTC m=+0.312379626,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-25.ec2.internal,}" Jul 15 19:14:25.046647 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.046636 2568 policy_none.go:49] "None policy: Start" Jul 15 19:14:25.046685 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.046655 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 19:14:25.046685 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.046668 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 19:14:25.048675 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:25.048664 2568 policy_none.go:47] "Start" Jul 15 19:14:26.160854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.160839 2568 manager.go:337] "Starting Device Plugin manager" Jul 15 19:14:26.160943 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.160866 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 19:14:26.160943 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.160875 2568 server.go:87] "Starting device plugin registration server" Jul 15 19:14:26.161108 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.161095 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 19:14:26.161157 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.161110 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 19:14:26.161418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.161239 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 19:14:26.161418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.161330 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 19:14:26.161418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.161338 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 19:14:26.161819 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.161800 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 19:14:26.161887 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.161849 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.221285 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.221261 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 19:14:26.223392 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.222397 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 19:14:26.223392 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.222436 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 19:14:26.223392 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.222453 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 19:14:26.223392 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.222481 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 19:14:26.224870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.224847 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:14:26.261854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.261837 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:14:26.262679 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.262666 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:14:26.262735 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.262690 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:14:26.262735 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.262699 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:14:26.262735 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.262719 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.270846 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.270832 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.270888 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.270852 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-25.ec2.internal\": node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.282865 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.282819 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.323476 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.323456 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal"] Jul 15 19:14:26.323541 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.323509 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:14:26.324176 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.324163 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:14:26.324239 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.324188 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:14:26.324239 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.324197 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:14:26.326483 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.326472 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:14:26.326600 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.326588 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.326638 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.326613 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:14:26.327102 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.327085 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:14:26.327179 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.327108 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:14:26.327179 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.327119 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:14:26.327279 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.327185 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:14:26.327279 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.327216 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:14:26.327279 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.327232 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:14:26.329417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.329402 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.329471 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.329430 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 19:14:26.330097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.330080 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientMemory" Jul 15 19:14:26.330157 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.330114 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 19:14:26.330157 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.330128 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeHasSufficientPID" Jul 15 19:14:26.342863 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.342846 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-25.ec2.internal\" not found" node="ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.346977 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.346961 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-25.ec2.internal\" not found" node="ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.382949 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.382936 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.404333 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.404315 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aefba3ae57e796fcd307ce94d0d35ce8-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.ec2.internal\" (UID: \"aefba3ae57e796fcd307ce94d0d35ce8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.404400 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.404339 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55a9be521bebb6a8869d125a3c51b1c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal\" (UID: \"55a9be521bebb6a8869d125a3c51b1c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.404400 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.404376 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55a9be521bebb6a8869d125a3c51b1c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal\" (UID: \"55a9be521bebb6a8869d125a3c51b1c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.483004 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.482982 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.505366 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.505334 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55a9be521bebb6a8869d125a3c51b1c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal\" (UID: \"55a9be521bebb6a8869d125a3c51b1c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.505434 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.505365 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/55a9be521bebb6a8869d125a3c51b1c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal\" (UID: \"55a9be521bebb6a8869d125a3c51b1c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.505434 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.505421 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55a9be521bebb6a8869d125a3c51b1c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal\" (UID: \"55a9be521bebb6a8869d125a3c51b1c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.505536 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.505447 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aefba3ae57e796fcd307ce94d0d35ce8-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.ec2.internal\" (UID: \"aefba3ae57e796fcd307ce94d0d35ce8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.505536 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.505493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55a9be521bebb6a8869d125a3c51b1c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal\" (UID: \"55a9be521bebb6a8869d125a3c51b1c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.505536 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.505497 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/aefba3ae57e796fcd307ce94d0d35ce8-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.ec2.internal\" (UID: \"aefba3ae57e796fcd307ce94d0d35ce8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.583522 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.583479 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.645091 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.645067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.650053 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:26.649893 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" Jul 15 19:14:26.684464 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.684447 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.784941 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.784919 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.885411 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.885372 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:26.985844 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:26.985826 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:27.049319 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.049293 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 19:14:27.049671 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.049441 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 19:14:27.075377 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.075360 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:14:27.086285 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:27.086267 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-25.ec2.internal\" not found" Jul 15 19:14:27.121988 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.121961 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 19:09:26 +0000 UTC" deadline="2028-01-04 04:39:39.078393548 +0000 UTC" Jul 15 19:14:27.121988 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.121986 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12897h25m11.95641032s" Jul 15 19:14:27.158733 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.158691 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:14:27.203048 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.203020 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" Jul 15 19:14:27.208392 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:27.208365 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaefba3ae57e796fcd307ce94d0d35ce8.slice/crio-0c89197d265abcbcf481cbd454b1092150f00f136a1bcf7e20608c38ce57dc3f WatchSource:0}: Error finding container 0c89197d265abcbcf481cbd454b1092150f00f136a1bcf7e20608c38ce57dc3f: Status 404 returned error can't find the container with id 0c89197d265abcbcf481cbd454b1092150f00f136a1bcf7e20608c38ce57dc3f Jul 15 19:14:27.213855 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.213843 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:14:27.214566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.214554 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:14:27.215942 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.215914 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" Jul 15 19:14:27.224936 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.224901 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" event={"ID":"aefba3ae57e796fcd307ce94d0d35ce8","Type":"ContainerStarted","Data":"0c89197d265abcbcf481cbd454b1092150f00f136a1bcf7e20608c38ce57dc3f"} Jul 15 19:14:27.225367 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.225337 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 19:14:27.321934 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:27.321907 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a9be521bebb6a8869d125a3c51b1c6.slice/crio-77070c7ec16d6f163490ac6f33e02cadd72ea280017aa3a6eaa4a44226ad33d9 WatchSource:0}: Error finding container 77070c7ec16d6f163490ac6f33e02cadd72ea280017aa3a6eaa4a44226ad33d9: Status 404 returned error can't find the container with id 77070c7ec16d6f163490ac6f33e02cadd72ea280017aa3a6eaa4a44226ad33d9 Jul 15 19:14:27.353060 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:27.353037 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:14:28.086282 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.086251 2568 apiserver.go:52] "Watching apiserver" Jul 15 19:14:28.094402 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.094371 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 19:14:28.094955 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.094933 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-v4q6d","openshift-dns/node-resolver-pwqs8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal","openshift-multus/multus-cd54h","openshift-multus/network-metrics-daemon-7jgjx","openshift-network-operator/iptables-alerter-447lb","kube-system/global-pull-secret-syncer-64nd4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl","openshift-image-registry/node-ca-4r98t","openshift-multus/multus-additional-cni-plugins-vb6dv","openshift-network-diagnostics/network-check-target-2rf4z","openshift-ovn-kubernetes/ovnkube-node-kft5f","kube-system/konnectivity-agent-59pm7","kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal"] Jul 15 19:14:28.097418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.097398 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.097525 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.097496 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:28.099624 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.099606 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.102037 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.102014 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.103786 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.103769 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 19:14:28.104281 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.104265 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.106402 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.106383 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 19:14:28.106651 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.106636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:28.106726 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.106695 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:28.108671 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.108650 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.111140 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.111119 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.113151 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113130 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.113788 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113766 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c19c9b26-f262-4a96-983b-25677ed981c3-agent-certs\") pod \"konnectivity-agent-59pm7\" (UID: \"c19c9b26-f262-4a96-983b-25677ed981c3\") " pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.113868 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113802 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.113868 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-device-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.113868 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113851 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-etc-selinux\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.114030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fng4m\" (UniqueName: \"kubernetes.io/projected/aa31f668-2e90-483d-adae-e020af3876e9-kube-api-access-fng4m\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113900 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24w8\" (UniqueName: \"kubernetes.io/projected/b5e1822c-c735-40eb-b9e0-48365f098ed8-kube-api-access-f24w8\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.114030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113926 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-cnibin\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa31f668-2e90-483d-adae-e020af3876e9-cni-binary-copy\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113974 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-hostroot\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.113998 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa31f668-2e90-483d-adae-e020af3876e9-multus-daemon-config\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-etc-kubernetes\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114053 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-multus-certs\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114085 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c19c9b26-f262-4a96-983b-25677ed981c3-konnectivity-ca\") pod \"konnectivity-agent-59pm7\" (UID: \"c19c9b26-f262-4a96-983b-25677ed981c3\") " pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-system-cni-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114132 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-os-release\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114157 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-cni-multus\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-socket-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-sys-fs\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114234 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-socket-dir-parent\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-cni-bin\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114368 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-registration-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-netns\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114426 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-kubelet\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-conf-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114488 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aa8fab3-6ea1-41fd-9ed6-182447618470-kubelet-config\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114523 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-cni-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-k8s-cni-cncf-io\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aa8fab3-6ea1-41fd-9ed6-182447618470-dbus\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.114854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.114609 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.115702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.115364 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 19:14:28.115702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.115417 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.117909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.117852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.122030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.122000 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 19:14:28.122385 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.122366 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:14:28.122741 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.122703 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:28.122829 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.122776 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:28.124414 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.124390 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t9ds6\"" Jul 15 19:14:28.124798 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.124778 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 19:14:28.125062 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.125046 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 19:14:28.125459 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.125440 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 19:14:28.125693 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.125673 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hf7f7\"" Jul 15 19:14:28.126119 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126093 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 19:14:28.126298 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126279 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126433 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126432 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126623 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126643 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126646 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126727 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126809 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.126875 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.127122 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ds4fg\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.127145 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f6vkg\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.127159 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.127263 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 19:14:28.127568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.127571 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jn9lv\"" Jul 15 19:14:28.128139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.127614 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mqnx7\"" Jul 15 19:14:28.128980 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.128939 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.130895 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.130868 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 19:14:28.131748 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.131728 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c58dr\"" Jul 15 19:14:28.132189 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.132171 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hlbvl\"" Jul 15 19:14:28.138844 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.138823 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 19:14:28.138996 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.138977 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 19:14:28.139063 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.139047 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 19:14:28.139148 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.139132 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 19:14:28.139216 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.139199 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 19:14:28.139273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.139221 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 19:14:28.139273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.139251 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 19:14:28.139386 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.139313 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 19:14:28.139386 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.139330 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wftx6\"" Jul 15 19:14:28.154001 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.153981 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkrbn" Jul 15 19:14:28.163064 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.163042 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nkrbn" Jul 15 19:14:28.205087 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.205066 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 19:14:28.215219 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-node-log\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.215398 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215230 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-log-socket\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.215398 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215257 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-cnibin\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.215398 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa31f668-2e90-483d-adae-e020af3876e9-cni-binary-copy\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.215398 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5bfb36a-68bf-4fdc-9689-d71008500ce6-host-slash\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.215398 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j29d4\" (UniqueName: \"kubernetes.io/projected/4f396e78-e744-4180-99ab-5e6b831458a1-kube-api-access-j29d4\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.215398 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215394 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902c3290-49b2-443b-a898-e758ae8dfa04-tmp-dir\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215419 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215446 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215457 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-cnibin\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215474 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-hostroot\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215534 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa31f668-2e90-483d-adae-e020af3876e9-multus-daemon-config\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/902c3290-49b2-443b-a898-e758ae8dfa04-hosts-file\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215587 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd7sr\" (UniqueName: \"kubernetes.io/projected/b9984863-de86-49c4-b3bc-352f5eb74188-kube-api-access-dd7sr\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-hostroot\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215621 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-run\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.215702 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c19c9b26-f262-4a96-983b-25677ed981c3-konnectivity-ca\") pod \"konnectivity-agent-59pm7\" (UID: \"c19c9b26-f262-4a96-983b-25677ed981c3\") " pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-cni-multus\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrwz\" (UniqueName: \"kubernetes.io/projected/a5bfb36a-68bf-4fdc-9689-d71008500ce6-kube-api-access-qsrwz\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215774 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f396e78-e744-4180-99ab-5e6b831458a1-host\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215778 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-cni-multus\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215798 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-os-release\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-socket-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-sys-fs\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-socket-dir-parent\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-sys\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-etc-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-socket-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-ovnkube-config\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215962 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-ovnkube-script-lib\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.215982 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-socket-dir-parent\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216007 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa31f668-2e90-483d-adae-e020af3876e9-cni-binary-copy\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216040 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-registration-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.216240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216053 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aa31f668-2e90-483d-adae-e020af3876e9-multus-daemon-config\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216069 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-sys-fs\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-system-cni-dir\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216117 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c19c9b26-f262-4a96-983b-25677ed981c3-konnectivity-ca\") pod \"konnectivity-agent-59pm7\" (UID: \"c19c9b26-f262-4a96-983b-25677ed981c3\") " pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-systemd-units\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-cni-bin\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216176 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-registration-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-env-overrides\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-cnibin\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-tuned\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216308 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-kubelet\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216335 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216379 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77566204-85b2-4542-bedd-5181138fec2e-ovn-node-metrics-cert\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysctl-d\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216423 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-run-netns\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217116 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c19c9b26-f262-4a96-983b-25677ed981c3-agent-certs\") pod \"konnectivity-agent-59pm7\" (UID: \"c19c9b26-f262-4a96-983b-25677ed981c3\") " pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-device-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216589 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-etc-selinux\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216616 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-systemd\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24w8\" (UniqueName: \"kubernetes.io/projected/b5e1822c-c735-40eb-b9e0-48365f098ed8-kube-api-access-f24w8\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216666 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-lib-modules\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216692 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-cni-netd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216717 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfvd\" (UniqueName: \"kubernetes.io/projected/77566204-85b2-4542-bedd-5181138fec2e-kube-api-access-8wfvd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216742 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-etc-kubernetes\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-var-lib-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216788 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-multus-certs\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216810 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-system-cni-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-os-release\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216831 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-host\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.217898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-tmp\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-cni-bin\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rl5\" (UniqueName: \"kubernetes.io/projected/902c3290-49b2-443b-a898-e758ae8dfa04-kube-api-access-j9rl5\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.216995 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpb4x\" (UniqueName: \"kubernetes.io/projected/3f80dfca-255e-4cce-9268-6aed4964c472-kube-api-access-rpb4x\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217051 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysctl-conf\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-systemd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217097 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-ovn\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-multus-certs\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217195 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-cni-bin\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217240 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-system-cni-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217332 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-etc-selinux\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b5e1822c-c735-40eb-b9e0-48365f098ed8-device-dir\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217430 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-etc-kubernetes\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-netns\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.218734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217473 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-kubelet\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217523 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-conf-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aa8fab3-6ea1-41fd-9ed6-182447618470-kubelet-config\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f396e78-e744-4180-99ab-5e6b831458a1-serviceca\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-os-release\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-cni-binary-copy\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217656 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-var-lib-kubelet\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217683 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-netns\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217708 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-cni-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217728 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-var-lib-kubelet\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217735 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-k8s-cni-cncf-io\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217769 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aa8fab3-6ea1-41fd-9ed6-182447618470-dbus\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217823 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5bfb36a-68bf-4fdc-9689-d71008500ce6-iptables-alerter-script\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-modprobe-d\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217854 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-cni-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8whvz\" (UniqueName: \"kubernetes.io/projected/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-kube-api-access-8whvz\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.219602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217897 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-host-run-k8s-cni-cncf-io\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217900 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fng4m\" (UniqueName: \"kubernetes.io/projected/aa31f668-2e90-483d-adae-e020af3876e9-kube-api-access-fng4m\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.217932 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysconfig\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.218079 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-kubernetes\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.218109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-slash\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.218149 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6aa8fab3-6ea1-41fd-9ed6-182447618470-dbus\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.218187 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa31f668-2e90-483d-adae-e020af3876e9-multus-conf-dir\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.218218 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6aa8fab3-6ea1-41fd-9ed6-182447618470-kubelet-config\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.218258 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:28.220410 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.218323 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret podName:6aa8fab3-6ea1-41fd-9ed6-182447618470 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:28.718295962 +0000 UTC m=+2.909173763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret") pod "global-pull-secret-syncer-64nd4" (UID: "6aa8fab3-6ea1-41fd-9ed6-182447618470") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:28.222647 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.222628 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c19c9b26-f262-4a96-983b-25677ed981c3-agent-certs\") pod \"konnectivity-agent-59pm7\" (UID: \"c19c9b26-f262-4a96-983b-25677ed981c3\") " pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.227915 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.227879 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" event={"ID":"55a9be521bebb6a8869d125a3c51b1c6","Type":"ContainerStarted","Data":"77070c7ec16d6f163490ac6f33e02cadd72ea280017aa3a6eaa4a44226ad33d9"} Jul 15 19:14:28.231813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.231791 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fng4m\" (UniqueName: \"kubernetes.io/projected/aa31f668-2e90-483d-adae-e020af3876e9-kube-api-access-fng4m\") pod \"multus-cd54h\" (UID: \"aa31f668-2e90-483d-adae-e020af3876e9\") " pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.232984 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.232427 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24w8\" (UniqueName: \"kubernetes.io/projected/b5e1822c-c735-40eb-b9e0-48365f098ed8-kube-api-access-f24w8\") pod \"aws-ebs-csi-driver-node-d6qxl\" (UID: \"b5e1822c-c735-40eb-b9e0-48365f098ed8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.318393 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318362 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-env-overrides\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318548 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318424 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-cnibin\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.318548 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318449 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-tuned\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.318548 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318473 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-kubelet\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318548 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77566204-85b2-4542-bedd-5181138fec2e-ovn-node-metrics-cert\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysctl-d\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-run-netns\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-systemd\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-lib-modules\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-cni-netd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfvd\" (UniqueName: \"kubernetes.io/projected/77566204-85b2-4542-bedd-5181138fec2e-kube-api-access-8wfvd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-var-lib-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.318766 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-host\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318786 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-tmp\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318808 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318816 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-env-overrides\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rl5\" (UniqueName: \"kubernetes.io/projected/902c3290-49b2-443b-a898-e758ae8dfa04-kube-api-access-j9rl5\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318879 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-run-ovn-kubernetes\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-cnibin\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.318919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpb4x\" (UniqueName: \"kubernetes.io/projected/3f80dfca-255e-4cce-9268-6aed4964c472-kube-api-access-rpb4x\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319086 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-var-lib-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319141 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-host\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.319240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319188 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-systemd\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.319729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319316 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-lib-modules\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.319729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319376 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-cni-netd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319473 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysctl-d\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.319729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319510 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-kubelet\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319544 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319957 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319922 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-run-netns\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.319957 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319923 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:28.320055 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319973 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320055 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.319973 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysctl-conf\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320055 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320015 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-systemd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320055 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320040 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-ovn\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f396e78-e744-4180-99ab-5e6b831458a1-serviceca\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320090 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-cni-binary-copy\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysctl-conf\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-var-lib-kubelet\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5bfb36a-68bf-4fdc-9689-d71008500ce6-iptables-alerter-script\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.320185 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-modprobe-d\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-ovn\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320222 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8whvz\" (UniqueName: \"kubernetes.io/projected/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-kube-api-access-8whvz\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320176 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-var-lib-kubelet\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.320243 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs podName:b9984863-de86-49c4-b3bc-352f5eb74188 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:28.82022602 +0000 UTC m=+3.011103809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs") pod "network-metrics-daemon-7jgjx" (UID: "b9984863-de86-49c4-b3bc-352f5eb74188") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320308 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-run-systemd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysconfig\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320468 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-kubernetes\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320491 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-slash\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-node-log\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320569 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-log-socket\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5bfb36a-68bf-4fdc-9689-d71008500ce6-host-slash\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j29d4\" (UniqueName: \"kubernetes.io/projected/4f396e78-e744-4180-99ab-5e6b831458a1-kube-api-access-j29d4\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-modprobe-d\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320675 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5bfb36a-68bf-4fdc-9689-d71008500ce6-host-slash\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320675 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902c3290-49b2-443b-a898-e758ae8dfa04-tmp-dir\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.320718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/902c3290-49b2-443b-a898-e758ae8dfa04-hosts-file\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd7sr\" (UniqueName: \"kubernetes.io/projected/b9984863-de86-49c4-b3bc-352f5eb74188-kube-api-access-dd7sr\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-run\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrwz\" (UniqueName: \"kubernetes.io/projected/a5bfb36a-68bf-4fdc-9689-d71008500ce6-kube-api-access-qsrwz\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320924 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/902c3290-49b2-443b-a898-e758ae8dfa04-tmp-dir\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320932 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f396e78-e744-4180-99ab-5e6b831458a1-host\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.320958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-os-release\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-sys\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-etc-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-ovnkube-config\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321236 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-cni-binary-copy\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321297 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-sys\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-ovnkube-config\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321624 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-ovnkube-script-lib\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-system-cni-dir\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-systemd-units\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-cni-bin\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321760 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-system-cni-dir\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.321913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.321847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-cni-bin\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.322232 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322209 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f80dfca-255e-4cce-9268-6aed4964c472-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.322304 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322284 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-tmp\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.322368 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/77566204-85b2-4542-bedd-5181138fec2e-ovnkube-script-lib\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.322419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322378 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-kubernetes\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.322419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322396 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/902c3290-49b2-443b-a898-e758ae8dfa04-hosts-file\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.322481 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322445 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-host-slash\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.322535 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-node-log\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.322535 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.322601 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322546 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-sysconfig\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.322727 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322710 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-run\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.322763 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322751 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4f396e78-e744-4180-99ab-5e6b831458a1-host\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.322803 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322789 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-log-socket\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.322845 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-systemd-units\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.322880 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/77566204-85b2-4542-bedd-5181138fec2e-etc-openvswitch\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.322932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5bfb36a-68bf-4fdc-9689-d71008500ce6-iptables-alerter-script\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.322980 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322930 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3f80dfca-255e-4cce-9268-6aed4964c472-os-release\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.322980 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.322935 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4f396e78-e744-4180-99ab-5e6b831458a1-serviceca\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.324442 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.324423 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-etc-tuned\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.324831 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.324631 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/77566204-85b2-4542-bedd-5181138fec2e-ovn-node-metrics-cert\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.328325 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.328286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rl5\" (UniqueName: \"kubernetes.io/projected/902c3290-49b2-443b-a898-e758ae8dfa04-kube-api-access-j9rl5\") pod \"node-resolver-pwqs8\" (UID: \"902c3290-49b2-443b-a898-e758ae8dfa04\") " pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.329616 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.329596 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfvd\" (UniqueName: \"kubernetes.io/projected/77566204-85b2-4542-bedd-5181138fec2e-kube-api-access-8wfvd\") pod \"ovnkube-node-kft5f\" (UID: \"77566204-85b2-4542-bedd-5181138fec2e\") " pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.333399 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.333378 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:14:28.333399 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.333402 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:14:28.333556 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.333415 2568 projected.go:196] Error preparing data for projected volume kube-api-access-86k6n for pod openshift-network-diagnostics/network-check-target-2rf4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:28.333556 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.333479 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n podName:60ca1071-d2d6-496c-961e-23291347bd7a nodeName:}" failed. No retries permitted until 2026-07-15 19:14:28.833462796 +0000 UTC m=+3.024340630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-86k6n" (UniqueName: "kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n") pod "network-check-target-2rf4z" (UID: "60ca1071-d2d6-496c-961e-23291347bd7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:28.334181 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.334072 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8whvz\" (UniqueName: \"kubernetes.io/projected/e60a8abe-2c7b-49b9-a0de-19d5e0d74425-kube-api-access-8whvz\") pod \"tuned-v4q6d\" (UID: \"e60a8abe-2c7b-49b9-a0de-19d5e0d74425\") " pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.335552 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.335511 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrwz\" (UniqueName: \"kubernetes.io/projected/a5bfb36a-68bf-4fdc-9689-d71008500ce6-kube-api-access-qsrwz\") pod \"iptables-alerter-447lb\" (UID: \"a5bfb36a-68bf-4fdc-9689-d71008500ce6\") " pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.336375 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.336309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j29d4\" (UniqueName: \"kubernetes.io/projected/4f396e78-e744-4180-99ab-5e6b831458a1-kube-api-access-j29d4\") pod \"node-ca-4r98t\" (UID: \"4f396e78-e744-4180-99ab-5e6b831458a1\") " pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.336996 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.336825 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpb4x\" (UniqueName: \"kubernetes.io/projected/3f80dfca-255e-4cce-9268-6aed4964c472-kube-api-access-rpb4x\") pod \"multus-additional-cni-plugins-vb6dv\" (UID: \"3f80dfca-255e-4cce-9268-6aed4964c472\") " pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.337102 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.337037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd7sr\" (UniqueName: \"kubernetes.io/projected/b9984863-de86-49c4-b3bc-352f5eb74188-kube-api-access-dd7sr\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:28.390811 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.390786 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 19:14:28.415886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.415865 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:28.430524 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.430503 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" Jul 15 19:14:28.440212 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.440193 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cd54h" Jul 15 19:14:28.446815 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.446799 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-447lb" Jul 15 19:14:28.453383 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.453361 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4r98t" Jul 15 19:14:28.465911 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.465894 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" Jul 15 19:14:28.471412 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.471394 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pwqs8" Jul 15 19:14:28.479952 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.479936 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" Jul 15 19:14:28.485466 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.485449 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:28.724969 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.724891 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:28.725127 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.725051 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:28.725127 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.725116 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret podName:6aa8fab3-6ea1-41fd-9ed6-182447618470 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:29.725101081 +0000 UTC m=+3.915978866 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret") pod "global-pull-secret-syncer-64nd4" (UID: "6aa8fab3-6ea1-41fd-9ed6-182447618470") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:28.825972 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.825938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:28.826123 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.826105 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:28.826189 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.826172 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs podName:b9984863-de86-49c4-b3bc-352f5eb74188 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:29.826156151 +0000 UTC m=+4.017033935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs") pod "network-metrics-daemon-7jgjx" (UID: "b9984863-de86-49c4-b3bc-352f5eb74188") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:28.926389 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:28.926356 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:28.926566 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.926511 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:14:28.926566 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.926530 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:14:28.926566 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.926541 2568 projected.go:196] Error preparing data for projected volume kube-api-access-86k6n for pod openshift-network-diagnostics/network-check-target-2rf4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:28.926681 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:28.926594 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n podName:60ca1071-d2d6-496c-961e-23291347bd7a nodeName:}" failed. No retries permitted until 2026-07-15 19:14:29.926577153 +0000 UTC m=+4.117454941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-86k6n" (UniqueName: "kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n") pod "network-check-target-2rf4z" (UID: "60ca1071-d2d6-496c-961e-23291347bd7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:29.166932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:29.166841 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:09:28 +0000 UTC" deadline="2027-12-23 03:31:35.295670589 +0000 UTC" Jul 15 19:14:29.166932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:29.166872 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12608h17m6.128802131s" Jul 15 19:14:29.284671 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.284225 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bfb36a_68bf_4fdc_9689_d71008500ce6.slice/crio-0d0075ba28842a124355748852ed2b12c8b78356dcde39a8e8765de9dffe39d3 WatchSource:0}: Error finding container 0d0075ba28842a124355748852ed2b12c8b78356dcde39a8e8765de9dffe39d3: Status 404 returned error can't find the container with id 0d0075ba28842a124355748852ed2b12c8b78356dcde39a8e8765de9dffe39d3 Jul 15 19:14:29.288549 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.288524 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e1822c_c735_40eb_b9e0_48365f098ed8.slice/crio-1074ccec0a423048c01d4a8efc9ac062bae0bd473aa2505196a01d2dddbd83fa WatchSource:0}: Error finding container 1074ccec0a423048c01d4a8efc9ac062bae0bd473aa2505196a01d2dddbd83fa: Status 404 returned error can't find the container with id 1074ccec0a423048c01d4a8efc9ac062bae0bd473aa2505196a01d2dddbd83fa Jul 15 19:14:29.289453 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.289410 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f80dfca_255e_4cce_9268_6aed4964c472.slice/crio-b4597f4ab73f0aa595a632194534201738f3cd6ade51312e6fbd1d5ca4786ef5 WatchSource:0}: Error finding container b4597f4ab73f0aa595a632194534201738f3cd6ade51312e6fbd1d5ca4786ef5: Status 404 returned error can't find the container with id b4597f4ab73f0aa595a632194534201738f3cd6ade51312e6fbd1d5ca4786ef5 Jul 15 19:14:29.290983 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.290962 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19c9b26_f262_4a96_983b_25677ed981c3.slice/crio-47beb16bb2ec933d86ce9e91ed08e25cb44821162769345a12908f79e6b10dd0 WatchSource:0}: Error finding container 47beb16bb2ec933d86ce9e91ed08e25cb44821162769345a12908f79e6b10dd0: Status 404 returned error can't find the container with id 47beb16bb2ec933d86ce9e91ed08e25cb44821162769345a12908f79e6b10dd0 Jul 15 19:14:29.292374 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.292330 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60a8abe_2c7b_49b9_a0de_19d5e0d74425.slice/crio-4a537165e5d6d736ed1a27e58fa6fa2508ff46f8e00a0f43a8712c0aca4c3d02 WatchSource:0}: Error finding container 4a537165e5d6d736ed1a27e58fa6fa2508ff46f8e00a0f43a8712c0aca4c3d02: Status 404 returned error can't find the container with id 4a537165e5d6d736ed1a27e58fa6fa2508ff46f8e00a0f43a8712c0aca4c3d02 Jul 15 19:14:29.293015 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.292926 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77566204_85b2_4542_bedd_5181138fec2e.slice/crio-ab2166498936c79650e213ac54717a62ca587f4f9259e076c02799ead103dc2b WatchSource:0}: Error finding container ab2166498936c79650e213ac54717a62ca587f4f9259e076c02799ead103dc2b: Status 404 returned error can't find the container with id ab2166498936c79650e213ac54717a62ca587f4f9259e076c02799ead103dc2b Jul 15 19:14:29.316550 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.316523 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902c3290_49b2_443b_a898_e758ae8dfa04.slice/crio-689ceb028b51c918ecca649f6813595a0fab1e12209bc6781678945020b8f139 WatchSource:0}: Error finding container 689ceb028b51c918ecca649f6813595a0fab1e12209bc6781678945020b8f139: Status 404 returned error can't find the container with id 689ceb028b51c918ecca649f6813595a0fab1e12209bc6781678945020b8f139 Jul 15 19:14:29.317329 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.317301 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f396e78_e744_4180_99ab_5e6b831458a1.slice/crio-bb9a6ed5ffb910103ebd7eee79d000e96dabe9a3e98538d07d3f84be800002c7 WatchSource:0}: Error finding container bb9a6ed5ffb910103ebd7eee79d000e96dabe9a3e98538d07d3f84be800002c7: Status 404 returned error can't find the container with id bb9a6ed5ffb910103ebd7eee79d000e96dabe9a3e98538d07d3f84be800002c7 Jul 15 19:14:29.318407 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:14:29.318383 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa31f668_2e90_483d_adae_e020af3876e9.slice/crio-679787dd9a3a7b6cc2873c6e55ab2a6effd1ca496fdcc33ba80ee2fd4c088ac1 WatchSource:0}: Error finding container 679787dd9a3a7b6cc2873c6e55ab2a6effd1ca496fdcc33ba80ee2fd4c088ac1: Status 404 returned error can't find the container with id 679787dd9a3a7b6cc2873c6e55ab2a6effd1ca496fdcc33ba80ee2fd4c088ac1 Jul 15 19:14:29.731433 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:29.731334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:29.731578 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.731476 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:29.731578 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.731554 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret podName:6aa8fab3-6ea1-41fd-9ed6-182447618470 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:31.731534302 +0000 UTC m=+5.922412087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret") pod "global-pull-secret-syncer-64nd4" (UID: "6aa8fab3-6ea1-41fd-9ed6-182447618470") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:29.831909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:29.831877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:29.832088 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.832076 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:29.832158 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.832133 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs podName:b9984863-de86-49c4-b3bc-352f5eb74188 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:31.832116916 +0000 UTC m=+6.022994703 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs") pod "network-metrics-daemon-7jgjx" (UID: "b9984863-de86-49c4-b3bc-352f5eb74188") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:29.932806 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:29.932680 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:29.932984 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.932852 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:14:29.932984 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.932870 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:14:29.932984 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.932884 2568 projected.go:196] Error preparing data for projected volume kube-api-access-86k6n for pod openshift-network-diagnostics/network-check-target-2rf4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:29.932984 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:29.932939 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n podName:60ca1071-d2d6-496c-961e-23291347bd7a nodeName:}" failed. No retries permitted until 2026-07-15 19:14:31.932920944 +0000 UTC m=+6.123798741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-86k6n" (UniqueName: "kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n") pod "network-check-target-2rf4z" (UID: "60ca1071-d2d6-496c-961e-23291347bd7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:30.167291 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.167219 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 19:09:28 +0000 UTC" deadline="2028-02-03 01:10:54.993751534 +0000 UTC" Jul 15 19:14:30.167291 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.167287 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13613h56m24.826468651s" Jul 15 19:14:30.225945 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.225914 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:30.226134 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:30.226042 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:30.226637 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.226618 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:30.226740 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:30.226713 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:30.226805 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.226781 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:30.226876 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:30.226859 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:30.256681 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.256547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-59pm7" event={"ID":"c19c9b26-f262-4a96-983b-25677ed981c3","Type":"ContainerStarted","Data":"47beb16bb2ec933d86ce9e91ed08e25cb44821162769345a12908f79e6b10dd0"} Jul 15 19:14:30.258378 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.258327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerStarted","Data":"b4597f4ab73f0aa595a632194534201738f3cd6ade51312e6fbd1d5ca4786ef5"} Jul 15 19:14:30.279983 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.279899 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" event={"ID":"e60a8abe-2c7b-49b9-a0de-19d5e0d74425","Type":"ContainerStarted","Data":"4a537165e5d6d736ed1a27e58fa6fa2508ff46f8e00a0f43a8712c0aca4c3d02"} Jul 15 19:14:30.283059 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.283028 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" event={"ID":"b5e1822c-c735-40eb-b9e0-48365f098ed8","Type":"ContainerStarted","Data":"1074ccec0a423048c01d4a8efc9ac062bae0bd473aa2505196a01d2dddbd83fa"} Jul 15 19:14:30.319144 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.319101 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-447lb" event={"ID":"a5bfb36a-68bf-4fdc-9689-d71008500ce6","Type":"ContainerStarted","Data":"0d0075ba28842a124355748852ed2b12c8b78356dcde39a8e8765de9dffe39d3"} Jul 15 19:14:30.324027 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.323901 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" event={"ID":"aefba3ae57e796fcd307ce94d0d35ce8","Type":"ContainerStarted","Data":"01964080bfbf8b04a0623d24efc8f59d6acaa339c52141a461653a20518d2718"} Jul 15 19:14:30.352263 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.352228 2568 generic.go:356] "Generic (PLEG): container finished" podID="55a9be521bebb6a8869d125a3c51b1c6" containerID="ea6c493d06ba10c176672ab220da48fc028f47a6a77b0f40d7994b7a8486f0f0" exitCode=0 Jul 15 19:14:30.352454 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.352318 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" event={"ID":"55a9be521bebb6a8869d125a3c51b1c6","Type":"ContainerDied","Data":"ea6c493d06ba10c176672ab220da48fc028f47a6a77b0f40d7994b7a8486f0f0"} Jul 15 19:14:30.367303 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.367234 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cd54h" event={"ID":"aa31f668-2e90-483d-adae-e020af3876e9","Type":"ContainerStarted","Data":"679787dd9a3a7b6cc2873c6e55ab2a6effd1ca496fdcc33ba80ee2fd4c088ac1"} Jul 15 19:14:30.373334 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.372840 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.ec2.internal" podStartSLOduration=3.372817106 podStartE2EDuration="3.372817106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:30.340633476 +0000 UTC m=+4.531511302" watchObservedRunningTime="2026-07-15 19:14:30.372817106 +0000 UTC m=+4.563694912" Jul 15 19:14:30.397149 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.397026 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4r98t" event={"ID":"4f396e78-e744-4180-99ab-5e6b831458a1","Type":"ContainerStarted","Data":"bb9a6ed5ffb910103ebd7eee79d000e96dabe9a3e98538d07d3f84be800002c7"} Jul 15 19:14:30.409611 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.409513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwqs8" event={"ID":"902c3290-49b2-443b-a898-e758ae8dfa04","Type":"ContainerStarted","Data":"689ceb028b51c918ecca649f6813595a0fab1e12209bc6781678945020b8f139"} Jul 15 19:14:30.419669 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:30.419514 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"ab2166498936c79650e213ac54717a62ca587f4f9259e076c02799ead103dc2b"} Jul 15 19:14:31.450853 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:31.450751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" event={"ID":"55a9be521bebb6a8869d125a3c51b1c6","Type":"ContainerStarted","Data":"2ed32328eaf3e28f81846b674bdd181f91435f979bd660919deb0b5ef162e8a5"} Jul 15 19:14:31.469477 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:31.468370 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.ec2.internal" podStartSLOduration=4.468336203 podStartE2EDuration="4.468336203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:31.468021624 +0000 UTC m=+5.658899450" watchObservedRunningTime="2026-07-15 19:14:31.468336203 +0000 UTC m=+5.659214011" Jul 15 19:14:31.746311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:31.746227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:31.746504 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.746434 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:31.746587 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.746506 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret podName:6aa8fab3-6ea1-41fd-9ed6-182447618470 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:35.746483967 +0000 UTC m=+9.937361760 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret") pod "global-pull-secret-syncer-64nd4" (UID: "6aa8fab3-6ea1-41fd-9ed6-182447618470") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:31.847598 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:31.847563 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:31.847849 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.847814 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:31.847932 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.847887 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs podName:b9984863-de86-49c4-b3bc-352f5eb74188 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:35.847868929 +0000 UTC m=+10.038746725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs") pod "network-metrics-daemon-7jgjx" (UID: "b9984863-de86-49c4-b3bc-352f5eb74188") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:31.948786 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:31.948751 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:31.948950 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.948906 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:14:31.948950 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.948928 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:14:31.948950 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.948941 2568 projected.go:196] Error preparing data for projected volume kube-api-access-86k6n for pod openshift-network-diagnostics/network-check-target-2rf4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:31.949121 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:31.948995 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n podName:60ca1071-d2d6-496c-961e-23291347bd7a nodeName:}" failed. No retries permitted until 2026-07-15 19:14:35.948977974 +0000 UTC m=+10.139855773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-86k6n" (UniqueName: "kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n") pod "network-check-target-2rf4z" (UID: "60ca1071-d2d6-496c-961e-23291347bd7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:32.225752 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:32.225119 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:32.225752 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:32.225246 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:32.225752 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:32.225429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:32.225752 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:32.225526 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:32.225752 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:32.225583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:32.225752 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:32.225653 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:34.224048 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:34.223986 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:34.224553 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:34.224125 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:34.224553 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:34.224122 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:34.224553 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:34.224195 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:34.224553 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:34.224245 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:34.224553 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:34.224367 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:35.783380 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:35.783272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:35.783773 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.783473 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:35.783773 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.783548 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret podName:6aa8fab3-6ea1-41fd-9ed6-182447618470 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:43.783527502 +0000 UTC m=+17.974405302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret") pod "global-pull-secret-syncer-64nd4" (UID: "6aa8fab3-6ea1-41fd-9ed6-182447618470") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:35.883909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:35.883862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:35.884121 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.884044 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:35.884121 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.884117 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs podName:b9984863-de86-49c4-b3bc-352f5eb74188 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:43.884095868 +0000 UTC m=+18.074973667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs") pod "network-metrics-daemon-7jgjx" (UID: "b9984863-de86-49c4-b3bc-352f5eb74188") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:35.985107 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:35.985066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:35.985313 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.985227 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:14:35.985313 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.985249 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:14:35.985313 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.985260 2568 projected.go:196] Error preparing data for projected volume kube-api-access-86k6n for pod openshift-network-diagnostics/network-check-target-2rf4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:35.985540 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:35.985353 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n podName:60ca1071-d2d6-496c-961e-23291347bd7a nodeName:}" failed. No retries permitted until 2026-07-15 19:14:43.985325266 +0000 UTC m=+18.176203065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-86k6n" (UniqueName: "kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n") pod "network-check-target-2rf4z" (UID: "60ca1071-d2d6-496c-961e-23291347bd7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:36.224418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:36.224037 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:36.224418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:36.224138 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:36.224418 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:36.224137 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:36.224418 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:36.224220 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:36.224418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:36.224265 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:36.224418 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:36.224325 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:38.223025 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:38.222993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:38.223616 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:38.223126 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:38.223616 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:38.223330 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:38.223616 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:38.223571 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:38.223784 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:38.223568 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:38.223784 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:38.223688 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:40.223437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:40.223335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:40.223882 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:40.223358 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:40.223882 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:40.223479 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:40.223882 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:40.223580 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:40.223882 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:40.223358 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:40.223882 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:40.223691 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:42.222965 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:42.222930 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:42.222965 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:42.222951 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:42.223479 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:42.222930 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:42.223479 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:42.223058 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:42.223479 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:42.223131 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:42.223479 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:42.223217 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:43.854334 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:43.854292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:43.854853 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:43.854475 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:43.854853 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:43.854541 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret podName:6aa8fab3-6ea1-41fd-9ed6-182447618470 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:59.85452486 +0000 UTC m=+34.045402643 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret") pod "global-pull-secret-syncer-64nd4" (UID: "6aa8fab3-6ea1-41fd-9ed6-182447618470") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:43.955463 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:43.955431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:43.955622 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:43.955534 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:43.955622 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:43.955585 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs podName:b9984863-de86-49c4-b3bc-352f5eb74188 nodeName:}" failed. No retries permitted until 2026-07-15 19:14:59.955571704 +0000 UTC m=+34.146449502 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs") pod "network-metrics-daemon-7jgjx" (UID: "b9984863-de86-49c4-b3bc-352f5eb74188") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:44.055981 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:44.055938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:44.056162 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:44.056077 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:14:44.056162 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:44.056104 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:14:44.056162 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:44.056117 2568 projected.go:196] Error preparing data for projected volume kube-api-access-86k6n for pod openshift-network-diagnostics/network-check-target-2rf4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:44.056368 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:44.056183 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n podName:60ca1071-d2d6-496c-961e-23291347bd7a nodeName:}" failed. No retries permitted until 2026-07-15 19:15:00.056163531 +0000 UTC m=+34.247041331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-86k6n" (UniqueName: "kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n") pod "network-check-target-2rf4z" (UID: "60ca1071-d2d6-496c-961e-23291347bd7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:14:44.223492 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:44.223429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:44.223635 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:44.223429 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:44.223635 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:44.223519 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:44.223635 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:44.223537 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:44.223635 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:44.223617 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:44.223825 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:44.223685 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:46.223753 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:46.223727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:46.224187 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:46.223826 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:46.224187 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:46.223911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:46.224187 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:46.224079 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:46.224187 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:46.224132 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:46.224372 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:46.224236 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:48.223663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.223631 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:48.224057 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.223672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:48.224057 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.223631 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:48.224057 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:48.223738 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:48.224057 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:48.223811 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:48.224057 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:48.223873 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:48.486500 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.485752 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cd54h" event={"ID":"aa31f668-2e90-483d-adae-e020af3876e9","Type":"ContainerStarted","Data":"513a76edaa4744935de09d4f0a686348c4dd6d988e3559848583f40ef75f4bbd"} Jul 15 19:14:48.489324 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.489037 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4r98t" event={"ID":"4f396e78-e744-4180-99ab-5e6b831458a1","Type":"ContainerStarted","Data":"bea491745a6b11dfe941b0eb4b9127ac2bb708c23ff9d932144e472d5b536aee"} Jul 15 19:14:48.491262 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.491175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pwqs8" event={"ID":"902c3290-49b2-443b-a898-e758ae8dfa04","Type":"ContainerStarted","Data":"12da5448d13a00b22080d865b565376be61cb4a1e25f21ebd3ca51d4786be4c1"} Jul 15 19:14:48.494966 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.494591 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-59pm7" event={"ID":"c19c9b26-f262-4a96-983b-25677ed981c3","Type":"ContainerStarted","Data":"437f842663797473183ffea5f00b3f589a0e55ca2271e513e991b6b8703d0e1b"} Jul 15 19:14:48.496586 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.496565 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerStarted","Data":"1eeb2aef84fdec99da9cc148538ebb2127d2de117878270525cd752af41e12d5"} Jul 15 19:14:48.499121 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.498662 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" event={"ID":"e60a8abe-2c7b-49b9-a0de-19d5e0d74425","Type":"ContainerStarted","Data":"d49bf41c9ff296b0f2dbb136ecefc3358b4cd7e576d2df452d20495574bed6ad"} Jul 15 19:14:48.507279 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.507170 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cd54h" podStartSLOduration=3.521051199 podStartE2EDuration="22.507153219s" totalImagesPullingTime="18.98610202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:48.506746443 +0000 UTC m=+22.697624249" watchObservedRunningTime="2026-07-15 19:14:48.507153219 +0000 UTC m=+22.698031026" Jul 15 19:14:48.523003 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.522954 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4r98t" podStartSLOduration=11.876189228 podStartE2EDuration="22.522933378s" totalImagesPullingTime="10.64674415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:48.522064708 +0000 UTC m=+22.712942539" watchObservedRunningTime="2026-07-15 19:14:48.522933378 +0000 UTC m=+22.713811184" Jul 15 19:14:48.565075 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.564931 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-59pm7" podStartSLOduration=3.608550329 podStartE2EDuration="22.56491789s" totalImagesPullingTime="18.956367561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:48.564645899 +0000 UTC m=+22.755523715" watchObservedRunningTime="2026-07-15 19:14:48.56491789 +0000 UTC m=+22.755795694" Jul 15 19:14:48.585412 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.582994 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v4q6d" podStartSLOduration=3.624847561 podStartE2EDuration="22.582967265s" totalImagesPullingTime="18.958119704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:48.581730991 +0000 UTC m=+22.772608797" watchObservedRunningTime="2026-07-15 19:14:48.582967265 +0000 UTC m=+22.773845074" Jul 15 19:14:48.661167 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.661140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:48.739727 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.739704 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:48.842936 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:48.842911 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:49.104875 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.104845 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:49.105499 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.105476 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:49.122902 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.122861 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pwqs8" podStartSLOduration=4.1812152 podStartE2EDuration="23.122846515s" totalImagesPullingTime="18.941631315s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:48.598919212 +0000 UTC m=+22.789797016" watchObservedRunningTime="2026-07-15 19:14:49.122846515 +0000 UTC m=+23.313724322" Jul 15 19:14:49.504286 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.504258 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" event={"ID":"b5e1822c-c735-40eb-b9e0-48365f098ed8","Type":"ContainerStarted","Data":"4fde04adb5cd7952be29a96d75566db9ae5d35bbfb43d76ef2eac23078048aa3"} Jul 15 19:14:49.507015 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.506990 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"41423ed9dea675342347ebb82003bf9c7c96b6cbb92b42cfc8ac20ef28798b61"} Jul 15 19:14:49.507160 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.507067 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"9e81b2ac257ab0bca3e8ab1f2494a009eb40322c27d7746be7694637a5f29ea9"} Jul 15 19:14:49.507218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.507159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"564a1773a387c64355dfc7e5fb7ac013d9c9830505d5cf5661a1c1b38ae8be16"} Jul 15 19:14:49.507218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.507211 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"7f7b39573717d62ec4e5a7c268fe82eb9b246214ab97213cfb18ba561f8eaeed"} Jul 15 19:14:49.507323 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.507262 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"e1668a55f1cdcb10d68edebf428ab55a228dfee5f1184bec768d2f272c2dd817"} Jul 15 19:14:49.507323 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.507311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"883220d4e5769904330294e3a6c12d90bd5a0b2cb83c747b8cbe10d9d9b8e467"} Jul 15 19:14:49.508334 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.508313 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f80dfca-255e-4cce-9268-6aed4964c472" containerID="1eeb2aef84fdec99da9cc148538ebb2127d2de117878270525cd752af41e12d5" exitCode=0 Jul 15 19:14:49.508457 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.508390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerDied","Data":"1eeb2aef84fdec99da9cc148538ebb2127d2de117878270525cd752af41e12d5"} Jul 15 19:14:49.508614 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.508591 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:49.509647 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.509185 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-59pm7" Jul 15 19:14:49.602847 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.602782 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:49.749686 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:49.749664 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 19:14:50.171472 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.171362 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T19:14:49.74968081Z","UUID":"8d3e0e46-3526-4729-a627-ce5bd70c1f3c","Handler":null,"Name":"","Endpoint":""} Jul 15 19:14:50.173449 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.173427 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 19:14:50.173582 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.173515 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 19:14:50.222944 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.222918 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:50.222944 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.222940 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:50.223153 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.222958 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:50.223153 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:50.223051 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:50.223153 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:50.223118 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:50.223306 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:50.223199 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:50.513265 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.513186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" event={"ID":"b5e1822c-c735-40eb-b9e0-48365f098ed8","Type":"ContainerStarted","Data":"ec39400a6e212c6c79e5d59ef06a7d7d36974bfade06e861c96b9f614d4dcf6c"} Jul 15 19:14:50.515074 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.515041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-447lb" event={"ID":"a5bfb36a-68bf-4fdc-9689-d71008500ce6","Type":"ContainerStarted","Data":"29e177b79a9d6a9b60c72c7ebd170c868018c85a587db5e3ff396de81190295e"} Jul 15 19:14:50.533177 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.533132 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-447lb" podStartSLOduration=5.548216202 podStartE2EDuration="24.533119234s" totalImagesPullingTime="18.984903032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:50.533010455 +0000 UTC m=+24.723888264" watchObservedRunningTime="2026-07-15 19:14:50.533119234 +0000 UTC m=+24.723997038" Jul 15 19:14:50.659812 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.659774 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:50.761932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.761894 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:50.952777 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:50.952757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:14:51.521491 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:51.521211 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"dfce99501e55bcc464ff847e778e70814b8895067beb79296d1159b7324d11c6"} Jul 15 19:14:51.523399 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:51.523370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" event={"ID":"b5e1822c-c735-40eb-b9e0-48365f098ed8","Type":"ContainerStarted","Data":"02b522ab4f89191c0664e74e096adee1dc0227eb1ae82405a6456834b3f138f1"} Jul 15 19:14:51.550603 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:51.550551 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d6qxl" podStartSLOduration=4.404108416 podStartE2EDuration="25.550532075s" totalImagesPullingTime="21.146423659s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:51.548564704 +0000 UTC m=+25.739442510" watchObservedRunningTime="2026-07-15 19:14:51.550532075 +0000 UTC m=+25.741409881" Jul 15 19:14:52.223287 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:52.223257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:52.223473 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:52.223259 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:52.223473 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:52.223401 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:52.223473 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:52.223257 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:52.223638 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:52.223485 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:52.223638 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:52.223564 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:54.223572 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.223417 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:54.224141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.223426 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:54.224141 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:54.223662 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:54.224141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.223427 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:54.224141 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:54.223740 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:54.224141 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:54.223804 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:54.533439 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.533337 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" event={"ID":"77566204-85b2-4542-bedd-5181138fec2e","Type":"ContainerStarted","Data":"3c3ae62067b1c4a8a99a610dec828d2f8763790e8106d4cece0066a5e0979589"} Jul 15 19:14:54.533602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.533558 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:54.535094 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.535066 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f80dfca-255e-4cce-9268-6aed4964c472" containerID="4fa1797e2f747917b7bed4e7e10dc94bb6d566c937aa07b16731173f29fb94ae" exitCode=0 Jul 15 19:14:54.535220 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.535119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerDied","Data":"4fa1797e2f747917b7bed4e7e10dc94bb6d566c937aa07b16731173f29fb94ae"} Jul 15 19:14:54.549053 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.549033 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:54.562169 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:54.562128 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" podStartSLOduration=9.48363723 podStartE2EDuration="28.56211605s" totalImagesPullingTime="19.07847882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:14:54.561663869 +0000 UTC m=+28.752541887" watchObservedRunningTime="2026-07-15 19:14:54.56211605 +0000 UTC m=+28.752993855" Jul 15 19:14:55.537834 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.537808 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:55.538322 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.537899 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:55.551445 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.551424 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:14:55.967563 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.967390 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2rf4z"] Jul 15 19:14:55.967761 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.967646 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:55.967761 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:55.967721 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:55.968061 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.968035 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7jgjx"] Jul 15 19:14:55.968167 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.968136 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:55.968228 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:55.968212 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:55.980450 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.980425 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-64nd4"] Jul 15 19:14:55.980548 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:55.980506 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:55.980616 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:55.980579 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:56.541069 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:56.541033 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f80dfca-255e-4cce-9268-6aed4964c472" containerID="80328965445e3826ae7247aab14e05f60133d68e17c39bda5e07ce4a94c9663b" exitCode=0 Jul 15 19:14:56.541455 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:56.541117 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerDied","Data":"80328965445e3826ae7247aab14e05f60133d68e17c39bda5e07ce4a94c9663b"} Jul 15 19:14:57.223213 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:57.223177 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:57.223385 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:57.223275 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:58.223641 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:58.223611 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:14:58.223641 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:58.223647 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:58.224296 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:58.223731 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:14:58.224296 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:58.223818 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:14:58.548879 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:58.548850 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f80dfca-255e-4cce-9268-6aed4964c472" containerID="46cfc0763a3f11744c9710ccfd9af8532d3391dfbcaecfd656a58ca98c4c1a72" exitCode=0 Jul 15 19:14:58.549031 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:58.548892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerDied","Data":"46cfc0763a3f11744c9710ccfd9af8532d3391dfbcaecfd656a58ca98c4c1a72"} Jul 15 19:14:59.223465 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:59.223433 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:59.223639 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:59.223567 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-64nd4" podUID="6aa8fab3-6ea1-41fd-9ed6-182447618470" Jul 15 19:14:59.877975 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:59.877918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:14:59.878426 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:59.878070 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:59.878426 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:59.878133 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret podName:6aa8fab3-6ea1-41fd-9ed6-182447618470 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:31.878117119 +0000 UTC m=+66.068994901 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret") pod "global-pull-secret-syncer-64nd4" (UID: "6aa8fab3-6ea1-41fd-9ed6-182447618470") : object "kube-system"/"original-pull-secret" not registered Jul 15 19:14:59.978980 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:59.978948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:14:59.979162 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:59.979121 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:59.979236 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:14:59.979193 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs podName:b9984863-de86-49c4-b3bc-352f5eb74188 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:31.979177866 +0000 UTC m=+66.170055648 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs") pod "network-metrics-daemon-7jgjx" (UID: "b9984863-de86-49c4-b3bc-352f5eb74188") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 19:14:59.994988 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:14:59.994954 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:00.080293 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.080265 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:15:00.080480 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:00.080455 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 19:15:00.080480 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:00.080479 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 19:15:00.080556 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:00.080490 2568 projected.go:196] Error preparing data for projected volume kube-api-access-86k6n for pod openshift-network-diagnostics/network-check-target-2rf4z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:15:00.080556 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:00.080542 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n podName:60ca1071-d2d6-496c-961e-23291347bd7a nodeName:}" failed. No retries permitted until 2026-07-15 19:15:32.080526953 +0000 UTC m=+66.271404738 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-86k6n" (UniqueName: "kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n") pod "network-check-target-2rf4z" (UID: "60ca1071-d2d6-496c-961e-23291347bd7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 19:15:00.125043 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.125011 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:00.211331 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.211221 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:00.223267 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.223238 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:15:00.223450 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.223375 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:15:00.223506 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:00.223477 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2rf4z" podUID="60ca1071-d2d6-496c-961e-23291347bd7a" Jul 15 19:15:00.223506 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:00.223375 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7jgjx" podUID="b9984863-de86-49c4-b3bc-352f5eb74188" Jul 15 19:15:00.276886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.276848 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:00.357083 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.357030 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:00.496674 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.496564 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:00.598052 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:00.598019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:01.046618 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.046530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:01.101205 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.101162 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:01.155896 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.155871 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-25.ec2.internal" event="NodeReady" Jul 15 19:15:01.156042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.156013 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 19:15:01.193494 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.193469 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79b98d9579-gjx2j"] Jul 15 19:15:01.198204 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.198185 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.201280 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.201173 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 19:15:01.201280 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.201201 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xd8s9\"" Jul 15 19:15:01.201469 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.201276 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 19:15:01.201469 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.201386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 19:15:01.205865 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.205843 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2rggc"] Jul 15 19:15:01.211035 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.209083 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 19:15:01.212641 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.212544 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kkt6d"] Jul 15 19:15:01.213160 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.213012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.215455 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.215435 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 19:15:01.215544 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.215442 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 19:15:01.215642 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.215501 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l2njj\"" Jul 15 19:15:01.216183 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.216165 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79b98d9579-gjx2j"] Jul 15 19:15:01.216289 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.216269 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.218588 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.218571 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 19:15:01.218735 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.218656 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vvjwq\"" Jul 15 19:15:01.218839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.218821 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 19:15:01.218967 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.218865 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 19:15:01.221262 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.221246 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2rggc"] Jul 15 19:15:01.222663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.222648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:15:01.223812 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.223793 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kkt6d"] Jul 15 19:15:01.225177 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.225119 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:15:01.288400 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5247a32b-bd1b-4c81-990f-5be7568c79d6-ca-trust-extracted\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.288400 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288398 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-installation-pull-secrets\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.288589 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288441 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrx5d\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-kube-api-access-nrx5d\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.288589 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-certificates\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.288589 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288514 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-trusted-ca\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.288589 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-bound-sa-token\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.288762 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-image-registry-private-configuration\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.288762 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.288654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389234 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-image-registry-private-configuration\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389234 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e5338b-132f-4529-9a5d-8d38e2671420-config-volume\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.389234 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389210 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvr5\" (UniqueName: \"kubernetes.io/projected/f9e5338b-132f-4529-9a5d-8d38e2671420-kube-api-access-hwvr5\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.389498 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389374 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389498 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389416 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3ca659-d65d-4006-96c9-6de961080e75-cert\") pod \"ingress-canary-kkt6d\" (UID: \"7f3ca659-d65d-4006-96c9-6de961080e75\") " pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.389498 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389476 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5247a32b-bd1b-4c81-990f-5be7568c79d6-ca-trust-extracted\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389644 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-installation-pull-secrets\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389644 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389534 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e5338b-132f-4529-9a5d-8d38e2671420-tmp-dir\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.389644 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389568 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrx5d\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-kube-api-access-nrx5d\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389644 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.389644 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.389514 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:15:01.389644 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.389630 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b98d9579-gjx2j: secret "image-registry-tls" not found Jul 15 19:15:01.389644 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-certificates\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389956 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389665 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-trusted-ca\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389956 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.389693 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls podName:5247a32b-bd1b-4c81-990f-5be7568c79d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:01.889668868 +0000 UTC m=+36.080546655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls") pod "image-registry-79b98d9579-gjx2j" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6") : secret "image-registry-tls" not found Jul 15 19:15:01.389956 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389725 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg4t2\" (UniqueName: \"kubernetes.io/projected/7f3ca659-d65d-4006-96c9-6de961080e75-kube-api-access-dg4t2\") pod \"ingress-canary-kkt6d\" (UID: \"7f3ca659-d65d-4006-96c9-6de961080e75\") " pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.389956 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389776 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-bound-sa-token\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.389956 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.389859 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5247a32b-bd1b-4c81-990f-5be7568c79d6-ca-trust-extracted\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.390530 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.390501 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-certificates\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.390665 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.390625 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-trusted-ca\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.394014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.393856 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-installation-pull-secrets\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.394096 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.393873 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-image-registry-private-configuration\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.399610 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.399567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-bound-sa-token\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.399728 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.399635 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrx5d\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-kube-api-access-nrx5d\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.409382 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.409339 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:01.490336 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.490300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg4t2\" (UniqueName: \"kubernetes.io/projected/7f3ca659-d65d-4006-96c9-6de961080e75-kube-api-access-dg4t2\") pod \"ingress-canary-kkt6d\" (UID: \"7f3ca659-d65d-4006-96c9-6de961080e75\") " pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.490509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.490385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e5338b-132f-4529-9a5d-8d38e2671420-config-volume\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.490509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.490413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvr5\" (UniqueName: \"kubernetes.io/projected/f9e5338b-132f-4529-9a5d-8d38e2671420-kube-api-access-hwvr5\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.490509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.490463 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3ca659-d65d-4006-96c9-6de961080e75-cert\") pod \"ingress-canary-kkt6d\" (UID: \"7f3ca659-d65d-4006-96c9-6de961080e75\") " pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.490669 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.490514 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e5338b-132f-4529-9a5d-8d38e2671420-tmp-dir\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.490669 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.490560 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.490766 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.490675 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:15:01.490766 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.490732 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls podName:f9e5338b-132f-4529-9a5d-8d38e2671420 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:01.990714459 +0000 UTC m=+36.181592242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls") pod "dns-default-2rggc" (UID: "f9e5338b-132f-4529-9a5d-8d38e2671420") : secret "dns-default-metrics-tls" not found Jul 15 19:15:01.490926 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.490891 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9e5338b-132f-4529-9a5d-8d38e2671420-tmp-dir\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.491062 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.491015 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f9e5338b-132f-4529-9a5d-8d38e2671420-config-volume\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.493081 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.493060 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f3ca659-d65d-4006-96c9-6de961080e75-cert\") pod \"ingress-canary-kkt6d\" (UID: \"7f3ca659-d65d-4006-96c9-6de961080e75\") " pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.500157 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.500132 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvr5\" (UniqueName: \"kubernetes.io/projected/f9e5338b-132f-4529-9a5d-8d38e2671420-kube-api-access-hwvr5\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.500251 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.500160 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg4t2\" (UniqueName: \"kubernetes.io/projected/7f3ca659-d65d-4006-96c9-6de961080e75-kube-api-access-dg4t2\") pod \"ingress-canary-kkt6d\" (UID: \"7f3ca659-d65d-4006-96c9-6de961080e75\") " pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.521615 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.521586 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:01.531742 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.531726 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kkt6d" Jul 15 19:15:01.894445 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.894400 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:01.894641 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.894553 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:15:01.894641 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.894571 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b98d9579-gjx2j: secret "image-registry-tls" not found Jul 15 19:15:01.894641 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.894628 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls podName:5247a32b-bd1b-4c81-990f-5be7568c79d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:02.894611681 +0000 UTC m=+37.085489484 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls") pod "image-registry-79b98d9579-gjx2j" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6") : secret "image-registry-tls" not found Jul 15 19:15:01.995502 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:01.995462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:01.995674 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.995612 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:15:01.995721 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:01.995679 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls podName:f9e5338b-132f-4529-9a5d-8d38e2671420 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:02.995660923 +0000 UTC m=+37.186538742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls") pod "dns-default-2rggc" (UID: "f9e5338b-132f-4529-9a5d-8d38e2671420") : secret "dns-default-metrics-tls" not found Jul 15 19:15:02.223040 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.222959 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:15:02.223511 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.222964 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:15:02.227028 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.227005 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sk5rd\"" Jul 15 19:15:02.227028 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.227026 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:15:02.227218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.227038 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:15:02.227218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.227091 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pjk9x\"" Jul 15 19:15:02.227218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.227093 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:15:02.572292 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.572269 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kkt6d"] Jul 15 19:15:02.575776 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:02.575755 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3ca659_d65d_4006_96c9_6de961080e75.slice/crio-00df556124f7cd843df2adcc4ea739ba293a866d2e19399ca1beb2eed83791a0 WatchSource:0}: Error finding container 00df556124f7cd843df2adcc4ea739ba293a866d2e19399ca1beb2eed83791a0: Status 404 returned error can't find the container with id 00df556124f7cd843df2adcc4ea739ba293a866d2e19399ca1beb2eed83791a0 Jul 15 19:15:02.902661 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:02.902633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:02.902769 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:02.902746 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:15:02.902769 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:02.902758 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b98d9579-gjx2j: secret "image-registry-tls" not found Jul 15 19:15:02.902834 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:02.902806 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls podName:5247a32b-bd1b-4c81-990f-5be7568c79d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:04.902792764 +0000 UTC m=+39.093670547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls") pod "image-registry-79b98d9579-gjx2j" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6") : secret "image-registry-tls" not found Jul 15 19:15:03.003139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:03.003114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:03.003239 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:03.003227 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:15:03.003277 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:03.003271 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls podName:f9e5338b-132f-4529-9a5d-8d38e2671420 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:05.003257925 +0000 UTC m=+39.194135707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls") pod "dns-default-2rggc" (UID: "f9e5338b-132f-4529-9a5d-8d38e2671420") : secret "dns-default-metrics-tls" not found Jul 15 19:15:03.562738 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:03.562704 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f80dfca-255e-4cce-9268-6aed4964c472" containerID="057d797df5d67c65e45ed1bac1950e3d0d54a5015737143caecc1e53ab7bd902" exitCode=0 Jul 15 19:15:03.563180 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:03.562785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerDied","Data":"057d797df5d67c65e45ed1bac1950e3d0d54a5015737143caecc1e53ab7bd902"} Jul 15 19:15:03.564032 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:03.563947 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kkt6d" event={"ID":"7f3ca659-d65d-4006-96c9-6de961080e75","Type":"ContainerStarted","Data":"00df556124f7cd843df2adcc4ea739ba293a866d2e19399ca1beb2eed83791a0"} Jul 15 19:15:04.569420 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:04.569225 2568 generic.go:356] "Generic (PLEG): container finished" podID="3f80dfca-255e-4cce-9268-6aed4964c472" containerID="74797e615940eee4ff41726428515b0fb3b9779d16b25855061ed6888bcbc476" exitCode=0 Jul 15 19:15:04.569813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:04.569298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerDied","Data":"74797e615940eee4ff41726428515b0fb3b9779d16b25855061ed6888bcbc476"} Jul 15 19:15:04.747489 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:04.747438 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:04.814936 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:04.814909 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:04.919699 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:04.919621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:04.919870 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:04.919788 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:15:04.919870 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:04.919809 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b98d9579-gjx2j: secret "image-registry-tls" not found Jul 15 19:15:04.919981 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:04.919872 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls podName:5247a32b-bd1b-4c81-990f-5be7568c79d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:08.919855223 +0000 UTC m=+43.110733009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls") pod "image-registry-79b98d9579-gjx2j" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6") : secret "image-registry-tls" not found Jul 15 19:15:05.020314 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.020283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:05.020513 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:05.020461 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:15:05.020562 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:05.020528 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls podName:f9e5338b-132f-4529-9a5d-8d38e2671420 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:09.020511108 +0000 UTC m=+43.211388896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls") pod "dns-default-2rggc" (UID: "f9e5338b-132f-4529-9a5d-8d38e2671420") : secret "dns-default-metrics-tls" not found Jul 15 19:15:05.575237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.575210 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" event={"ID":"3f80dfca-255e-4cce-9268-6aed4964c472","Type":"ContainerStarted","Data":"c8421898b57952a3bdb9f1e2cd01118bbde9a5a31739646ec3b5c3d323793987"} Jul 15 19:15:05.602952 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.602902 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vb6dv" podStartSLOduration=9.980500108 podStartE2EDuration="39.602888498s" totalImagesPullingTime="29.62238839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:05.601527878 +0000 UTC m=+39.792405715" watchObservedRunningTime="2026-07-15 19:15:05.602888498 +0000 UTC m=+39.793766303" Jul 15 19:15:05.604886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.604865 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-rp5lk"] Jul 15 19:15:05.624155 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.624131 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-rp5lk"] Jul 15 19:15:05.624270 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.624249 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:05.626729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.626708 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 15 19:15:05.626729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.626717 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jmrwq\"" Jul 15 19:15:05.626910 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.626718 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 15 19:15:05.724979 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.724946 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9db371c-b7b8-448b-b63c-3be20f40bb63-nginx-conf\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:05.724979 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.724982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:05.729436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.729409 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:05.789735 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.789662 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:05.825798 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.825766 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9db371c-b7b8-448b-b63c-3be20f40bb63-nginx-conf\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:05.825798 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.825799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:05.825952 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:05.825931 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:15:05.826008 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:05.825982 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert podName:a9db371c-b7b8-448b-b63c-3be20f40bb63 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:06.325967785 +0000 UTC m=+40.516845568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rp5lk" (UID: "a9db371c-b7b8-448b-b63c-3be20f40bb63") : secret "networking-console-plugin-cert" not found Jul 15 19:15:05.826405 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.826389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9db371c-b7b8-448b-b63c-3be20f40bb63-nginx-conf\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:05.928541 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.928492 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:05.984524 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:05.984497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:06.092827 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.091672 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:06.191750 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.191718 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:06.292543 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.292509 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:06.329943 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.329859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:06.330110 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:06.330011 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:15:06.330110 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:06.330076 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert podName:a9db371c-b7b8-448b-b63c-3be20f40bb63 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:07.330059686 +0000 UTC m=+41.520937469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rp5lk" (UID: "a9db371c-b7b8-448b-b63c-3be20f40bb63") : secret "networking-console-plugin-cert" not found Jul 15 19:15:06.579337 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.579300 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kkt6d" event={"ID":"7f3ca659-d65d-4006-96c9-6de961080e75","Type":"ContainerStarted","Data":"6e8650c4e3cd28c7d03c4d7dd1299f76d0afecdf884a0740602b19ac69e5610c"} Jul 15 19:15:06.596986 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.596895 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kkt6d" podStartSLOduration=2.191387487 podStartE2EDuration="5.596875116s" totalImagesPullingTime="3.405487629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:06.596143835 +0000 UTC m=+40.787021667" watchObservedRunningTime="2026-07-15 19:15:06.596875116 +0000 UTC m=+40.787752921" Jul 15 19:15:06.871871 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.871779 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:06.935048 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:06.935016 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:07.338191 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.338162 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:07.338339 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:07.338297 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:15:07.338423 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:07.338379 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert podName:a9db371c-b7b8-448b-b63c-3be20f40bb63 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:09.338337007 +0000 UTC m=+43.529214792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rp5lk" (UID: "a9db371c-b7b8-448b-b63c-3be20f40bb63") : secret "networking-console-plugin-cert" not found Jul 15 19:15:07.558628 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.558598 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:07.640064 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.639992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:07.645550 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.645526 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xhtm6"] Jul 15 19:15:07.679292 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.679271 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xhtm6"] Jul 15 19:15:07.679411 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.679396 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.681715 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.681694 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 19:15:07.682104 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.682091 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gg6xc\"" Jul 15 19:15:07.682184 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.682094 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 19:15:07.683089 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.683069 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 19:15:07.683162 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.683097 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 19:15:07.703202 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.703185 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:07.842264 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.842239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.842396 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.842280 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-crio-socket\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.842447 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.842399 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-874q8\" (UniqueName: \"kubernetes.io/projected/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-api-access-874q8\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.842447 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.842430 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-data-volume\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.842514 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.842475 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.943744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.943690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-874q8\" (UniqueName: \"kubernetes.io/projected/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-api-access-874q8\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.943744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.943720 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-data-volume\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.943744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.943737 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.943941 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.943768 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.943941 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.943803 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-crio-socket\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.943941 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:07.943848 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:15:07.944081 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:07.943930 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:15:07.944081 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:07.943944 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:08.443922038 +0000 UTC m=+42.634799838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : configmap "kube-rbac-proxy" not found Jul 15 19:15:07.944081 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.944009 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-crio-socket\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.944081 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:07.944015 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:08.443995291 +0000 UTC m=+42.634873075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : secret "insights-runtime-extractor-tls" not found Jul 15 19:15:07.944231 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.944096 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-data-volume\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:07.956501 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:07.956484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-874q8\" (UniqueName: \"kubernetes.io/projected/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-api-access-874q8\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:08.136620 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.136594 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:08.236849 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.236829 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:08.401594 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.401529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:08.448813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.448792 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:08.448883 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.448830 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:08.448919 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:08.448898 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:15:08.448950 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:08.448927 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:15:08.448985 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:08.448956 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:09.448942214 +0000 UTC m=+43.639820001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : configmap "kube-rbac-proxy" not found Jul 15 19:15:08.448985 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:08.448969 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:09.448963139 +0000 UTC m=+43.639840925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : secret "insights-runtime-extractor-tls" not found Jul 15 19:15:08.498745 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.498692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:08.541965 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.541943 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr"] Jul 15 19:15:08.562995 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.562970 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr"] Jul 15 19:15:08.563076 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.563067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" Jul 15 19:15:08.565451 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.565427 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-52vmm\"" Jul 15 19:15:08.565537 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.565464 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 15 19:15:08.566311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.566294 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 15 19:15:08.584502 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.584428 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:08.644841 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.644819 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:08.731892 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.731871 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:08.751050 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.751007 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/5205921e-eadc-4e23-aec2-9292f5239f10-kube-api-access-dj5v6\") pod \"migrator-7f59b54f6c-tqbvr\" (UID: \"5205921e-eadc-4e23-aec2-9292f5239f10\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" Jul 15 19:15:08.851571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.851550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/5205921e-eadc-4e23-aec2-9292f5239f10-kube-api-access-dj5v6\") pod \"migrator-7f59b54f6c-tqbvr\" (UID: \"5205921e-eadc-4e23-aec2-9292f5239f10\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" Jul 15 19:15:08.861042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.861012 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5v6\" (UniqueName: \"kubernetes.io/projected/5205921e-eadc-4e23-aec2-9292f5239f10-kube-api-access-dj5v6\") pod \"migrator-7f59b54f6c-tqbvr\" (UID: \"5205921e-eadc-4e23-aec2-9292f5239f10\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" Jul 15 19:15:08.871787 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.871769 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" Jul 15 19:15:08.952582 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:08.952540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:08.952729 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:08.952671 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:15:08.952729 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:08.952683 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b98d9579-gjx2j: secret "image-registry-tls" not found Jul 15 19:15:08.952729 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:08.952731 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls podName:5247a32b-bd1b-4c81-990f-5be7568c79d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:16.952716396 +0000 UTC m=+51.143594193 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls") pod "image-registry-79b98d9579-gjx2j" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6") : secret "image-registry-tls" not found Jul 15 19:15:09.005293 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:09.005267 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr"] Jul 15 19:15:09.007511 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:09.007476 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5205921e_eadc_4e23_aec2_9292f5239f10.slice/crio-897a99720c1e23c01700333b4feccb5e900d0a7a72e26b8e22d34138f1e24725 WatchSource:0}: Error finding container 897a99720c1e23c01700333b4feccb5e900d0a7a72e26b8e22d34138f1e24725: Status 404 returned error can't find the container with id 897a99720c1e23c01700333b4feccb5e900d0a7a72e26b8e22d34138f1e24725 Jul 15 19:15:09.040708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:09.040687 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pwqs8_902c3290-49b2-443b-a898-e758ae8dfa04/dns-node-resolver/0.log" Jul 15 19:15:09.053030 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:09.053006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:09.053143 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.053127 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:15:09.053219 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.053207 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls podName:f9e5338b-132f-4529-9a5d-8d38e2671420 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:17.053186676 +0000 UTC m=+51.244064472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls") pod "dns-default-2rggc" (UID: "f9e5338b-132f-4529-9a5d-8d38e2671420") : secret "dns-default-metrics-tls" not found Jul 15 19:15:09.355113 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:09.355050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:09.355235 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.355218 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:15:09.355311 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.355300 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert podName:a9db371c-b7b8-448b-b63c-3be20f40bb63 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:13.355278714 +0000 UTC m=+47.546156519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rp5lk" (UID: "a9db371c-b7b8-448b-b63c-3be20f40bb63") : secret "networking-console-plugin-cert" not found Jul 15 19:15:09.456208 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:09.456185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:09.456299 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:09.456242 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:09.456299 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.456271 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:15:09.456394 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.456325 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:11.456308693 +0000 UTC m=+45.647186476 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : configmap "kube-rbac-proxy" not found Jul 15 19:15:09.456394 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.456332 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:15:09.456460 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:09.456407 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:11.45639271 +0000 UTC m=+45.647270504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : secret "insights-runtime-extractor-tls" not found Jul 15 19:15:09.585559 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:09.585537 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" event={"ID":"5205921e-eadc-4e23-aec2-9292f5239f10","Type":"ContainerStarted","Data":"897a99720c1e23c01700333b4feccb5e900d0a7a72e26b8e22d34138f1e24725"} Jul 15 19:15:10.005414 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.005387 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-txxcs"] Jul 15 19:15:10.025440 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.025418 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-txxcs"] Jul 15 19:15:10.025555 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.025498 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.028212 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.028181 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 19:15:10.028335 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.028250 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cr55z\"" Jul 15 19:15:10.028837 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.028821 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 19:15:10.029084 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.029069 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 19:15:10.029130 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.029077 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 19:15:10.041210 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.041192 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4r98t_4f396e78-e744-4180-99ab-5e6b831458a1/node-ca/0.log" Jul 15 19:15:10.045842 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.045825 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:10.061072 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.061047 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1112e8ff-0020-4c2a-bc09-f995d360d15b-signing-cabundle\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.061168 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.061081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1112e8ff-0020-4c2a-bc09-f995d360d15b-signing-key\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.061168 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.061124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpw59\" (UniqueName: \"kubernetes.io/projected/1112e8ff-0020-4c2a-bc09-f995d360d15b-kube-api-access-gpw59\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.104784 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.104760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:10.161955 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.161928 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1112e8ff-0020-4c2a-bc09-f995d360d15b-signing-cabundle\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.162052 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.161959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1112e8ff-0020-4c2a-bc09-f995d360d15b-signing-key\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.162052 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.161999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpw59\" (UniqueName: \"kubernetes.io/projected/1112e8ff-0020-4c2a-bc09-f995d360d15b-kube-api-access-gpw59\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.162731 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.162714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1112e8ff-0020-4c2a-bc09-f995d360d15b-signing-cabundle\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.164263 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.164247 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1112e8ff-0020-4c2a-bc09-f995d360d15b-signing-key\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.170718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.170698 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpw59\" (UniqueName: \"kubernetes.io/projected/1112e8ff-0020-4c2a-bc09-f995d360d15b-kube-api-access-gpw59\") pod \"service-ca-79f68b9f4f-txxcs\" (UID: \"1112e8ff-0020-4c2a-bc09-f995d360d15b\") " pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.334502 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.334436 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" Jul 15 19:15:10.765441 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:10.765408 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-txxcs"] Jul 15 19:15:10.768129 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:10.768103 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1112e8ff_0020_4c2a_bc09_f995d360d15b.slice/crio-a791da43774db0d606cafa47f8fd2189feab5e9da61d1c1da14f917486563b9d WatchSource:0}: Error finding container a791da43774db0d606cafa47f8fd2189feab5e9da61d1c1da14f917486563b9d: Status 404 returned error can't find the container with id a791da43774db0d606cafa47f8fd2189feab5e9da61d1c1da14f917486563b9d Jul 15 19:15:11.041825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.041804 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kkt6d_7f3ca659-d65d-4006-96c9-6de961080e75/serve-healthcheck-canary/0.log" Jul 15 19:15:11.391790 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.391742 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:11.473987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.473951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:11.474181 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.474019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:11.474181 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:11.474088 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:15:11.474181 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:11.474162 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:15.474141707 +0000 UTC m=+49.665019492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : configmap "kube-rbac-proxy" not found Jul 15 19:15:11.474320 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:11.474213 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:15:11.474320 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:11.474275 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:15.474257165 +0000 UTC m=+49.665134965 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : secret "insights-runtime-extractor-tls" not found Jul 15 19:15:11.592634 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.592498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" event={"ID":"5205921e-eadc-4e23-aec2-9292f5239f10","Type":"ContainerStarted","Data":"c4e88cbd99b30cc753de153213dc398ab0f182312c43c9778e5f664f03fd5f5c"} Jul 15 19:15:11.592634 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.592592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" event={"ID":"5205921e-eadc-4e23-aec2-9292f5239f10","Type":"ContainerStarted","Data":"0dd1df438eee271c23eebf729a3caa7d931a14261e3d6c30364df8e2cf3e2617"} Jul 15 19:15:11.593851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.593819 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" event={"ID":"1112e8ff-0020-4c2a-bc09-f995d360d15b","Type":"ContainerStarted","Data":"a791da43774db0d606cafa47f8fd2189feab5e9da61d1c1da14f917486563b9d"} Jul 15 19:15:11.612698 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:11.612633 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-tqbvr" podStartSLOduration=1.647183889 podStartE2EDuration="3.612612547s" totalImagesPullingTime="1.965428658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:11.61102033 +0000 UTC m=+45.801898160" watchObservedRunningTime="2026-07-15 19:15:11.612612547 +0000 UTC m=+45.803490354" Jul 15 19:15:13.390681 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:13.390646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:13.391086 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:13.390780 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:15:13.391086 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:13.390839 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert podName:a9db371c-b7b8-448b-b63c-3be20f40bb63 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:21.390822423 +0000 UTC m=+55.581700225 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rp5lk" (UID: "a9db371c-b7b8-448b-b63c-3be20f40bb63") : secret "networking-console-plugin-cert" not found Jul 15 19:15:13.600438 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:13.600405 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" event={"ID":"1112e8ff-0020-4c2a-bc09-f995d360d15b","Type":"ContainerStarted","Data":"afba475ac1b9c23106e67f3cd2274f45f05491bfbe535ffabaec0e3e8288822d"} Jul 15 19:15:13.619470 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:13.619415 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-txxcs" podStartSLOduration=1.994661168 podStartE2EDuration="4.619393617s" totalImagesPullingTime="2.624732449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:13.619276236 +0000 UTC m=+47.810154065" watchObservedRunningTime="2026-07-15 19:15:13.619393617 +0000 UTC m=+47.810271424" Jul 15 19:15:14.049024 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:14.048989 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:15.505967 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:15.505931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:15.506509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:15.506034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:15.506509 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:15.506102 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 19:15:15.506509 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:15.506105 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:15:15.506509 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:15.506151 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:23.506137436 +0000 UTC m=+57.697015219 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : configmap "kube-rbac-proxy" not found Jul 15 19:15:15.506509 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:15.506164 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:23.506157387 +0000 UTC m=+57.697035170 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : secret "insights-runtime-extractor-tls" not found Jul 15 19:15:15.782275 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:15.782185 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:16.452022 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:16.451983 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:16.555860 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:16.555821 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:17.017673 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:17.017637 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") pod \"image-registry-79b98d9579-gjx2j\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:17.017853 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:17.017777 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 19:15:17.017853 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:17.017797 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79b98d9579-gjx2j: secret "image-registry-tls" not found Jul 15 19:15:17.017923 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:17.017860 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls podName:5247a32b-bd1b-4c81-990f-5be7568c79d6 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:33.017844298 +0000 UTC m=+67.208722080 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls") pod "image-registry-79b98d9579-gjx2j" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6") : secret "image-registry-tls" not found Jul 15 19:15:17.038707 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:17.038671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:17.118755 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:17.118725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:17.118926 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:17.118871 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 19:15:17.118966 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:17.118961 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls podName:f9e5338b-132f-4529-9a5d-8d38e2671420 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:33.118942927 +0000 UTC m=+67.309820710 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls") pod "dns-default-2rggc" (UID: "f9e5338b-132f-4529-9a5d-8d38e2671420") : secret "dns-default-metrics-tls" not found Jul 15 19:15:17.136286 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:17.136253 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:17.476399 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:17.476310 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:17.878418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:17.878327 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:18.106794 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:18.106759 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:18.281179 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:18.281150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:18.722906 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:18.722832 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:19.471650 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:19.471616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:19.572383 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:19.572336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:19.682420 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:19.682391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:20.075676 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:20.075649 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:20.279335 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:20.279299 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:20.337393 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:20.337315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:20.675730 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:20.675664 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:20.775734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:20.775701 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:21.279549 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:21.279522 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:21.453421 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:21.453384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:21.453567 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:21.453516 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 15 19:15:21.453612 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:21.453571 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert podName:a9db371c-b7b8-448b-b63c-3be20f40bb63 nodeName:}" failed. No retries permitted until 2026-07-15 19:15:37.453557015 +0000 UTC m=+71.644434801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-rp5lk" (UID: "a9db371c-b7b8-448b-b63c-3be20f40bb63") : secret "networking-console-plugin-cert" not found Jul 15 19:15:21.681579 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:21.681552 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:21.880649 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:21.880622 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:22.144490 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:22.144463 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:22.280571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:22.280550 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:22.875468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:22.875436 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:23.284251 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:23.284222 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:23.569680 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:23.569618 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:23.569680 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:23.569658 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:23.569852 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:23.569805 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 19:15:23.569889 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:23.569850 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls podName:c1759d7c-a121-4bc4-9d93-7c9e62deb83b nodeName:}" failed. No retries permitted until 2026-07-15 19:15:39.569838465 +0000 UTC m=+73.760716248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xhtm6" (UID: "c1759d7c-a121-4bc4-9d93-7c9e62deb83b") : secret "insights-runtime-extractor-tls" not found Jul 15 19:15:23.570096 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:23.570080 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:23.677798 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:23.677751 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:24.083574 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:24.083546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:24.280222 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:24.280190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:24.858409 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:24.858378 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:24.951984 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:24.951962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:25.284005 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:25.283978 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:25.886980 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:25.886959 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:26.075559 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:26.075533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:26.282830 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:26.282803 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:26.478146 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:26.478107 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:27.557255 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:27.557227 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kft5f" Jul 15 19:15:28.365722 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.365690 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp"] Jul 15 19:15:28.368818 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.368800 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7"] Jul 15 19:15:28.368941 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.368923 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.370942 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.370921 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.371729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.371712 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 15 19:15:28.372139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.372114 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 15 19:15:28.372139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.372141 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 15 19:15:28.372322 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.372159 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 15 19:15:28.373951 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.373931 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-w876d\"" Jul 15 19:15:28.374210 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.374197 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 15 19:15:28.380999 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.380977 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7"] Jul 15 19:15:28.384356 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.384319 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp"] Jul 15 19:15:28.404962 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.404867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073cd7ca-8452-4583-ba05-99ecec719f03-tmp\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.404962 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.404941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/073cd7ca-8452-4583-ba05-99ecec719f03-klusterlet-config\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.405137 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.404983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrx2c\" (UniqueName: \"kubernetes.io/projected/073cd7ca-8452-4583-ba05-99ecec719f03-kube-api-access-rrx2c\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.405137 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.405063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a07f72be-af31-43dc-9856-0b2e8c3bba48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-657994bc88-52lt7\" (UID: \"a07f72be-af31-43dc-9856-0b2e8c3bba48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.405137 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.405093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpbdw\" (UniqueName: \"kubernetes.io/projected/a07f72be-af31-43dc-9856-0b2e8c3bba48-kube-api-access-gpbdw\") pod \"managed-serviceaccount-addon-agent-657994bc88-52lt7\" (UID: \"a07f72be-af31-43dc-9856-0b2e8c3bba48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.487302 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.487279 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79b98d9579-gjx2j"] Jul 15 19:15:28.487474 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:15:28.487458 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" podUID="5247a32b-bd1b-4c81-990f-5be7568c79d6" Jul 15 19:15:28.492521 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.492499 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:28.506540 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.506522 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073cd7ca-8452-4583-ba05-99ecec719f03-tmp\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.506669 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.506574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/073cd7ca-8452-4583-ba05-99ecec719f03-klusterlet-config\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.506669 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.506603 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrx2c\" (UniqueName: \"kubernetes.io/projected/073cd7ca-8452-4583-ba05-99ecec719f03-kube-api-access-rrx2c\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.506669 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.506639 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a07f72be-af31-43dc-9856-0b2e8c3bba48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-657994bc88-52lt7\" (UID: \"a07f72be-af31-43dc-9856-0b2e8c3bba48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.506821 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.506680 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpbdw\" (UniqueName: \"kubernetes.io/projected/a07f72be-af31-43dc-9856-0b2e8c3bba48-kube-api-access-gpbdw\") pod \"managed-serviceaccount-addon-agent-657994bc88-52lt7\" (UID: \"a07f72be-af31-43dc-9856-0b2e8c3bba48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.506932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.506910 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/073cd7ca-8452-4583-ba05-99ecec719f03-tmp\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.509000 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.508977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a07f72be-af31-43dc-9856-0b2e8c3bba48-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-657994bc88-52lt7\" (UID: \"a07f72be-af31-43dc-9856-0b2e8c3bba48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.509080 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.509030 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/073cd7ca-8452-4583-ba05-99ecec719f03-klusterlet-config\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.526156 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.526137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrx2c\" (UniqueName: \"kubernetes.io/projected/073cd7ca-8452-4583-ba05-99ecec719f03-kube-api-access-rrx2c\") pod \"klusterlet-addon-workmgr-598d8687fc-2gncp\" (UID: \"073cd7ca-8452-4583-ba05-99ecec719f03\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.526235 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.526180 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpbdw\" (UniqueName: \"kubernetes.io/projected/a07f72be-af31-43dc-9856-0b2e8c3bba48-kube-api-access-gpbdw\") pod \"managed-serviceaccount-addon-agent-657994bc88-52lt7\" (UID: \"a07f72be-af31-43dc-9856-0b2e8c3bba48\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.530465 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.530444 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f6db4b9f-ktvp5"] Jul 15 19:15:28.533065 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.533049 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.544143 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.544122 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6db4b9f-ktvp5"] Jul 15 19:15:28.570430 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.570405 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:28.607711 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.607689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03bab594-36e8-4d64-8891-b440d7cda702-trusted-ca\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.607814 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.607794 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwzz9\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-kube-api-access-gwzz9\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.607863 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.607831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03bab594-36e8-4d64-8891-b440d7cda702-ca-trust-extracted\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.607863 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.607848 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03bab594-36e8-4d64-8891-b440d7cda702-installation-pull-secrets\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.607943 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.607877 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03bab594-36e8-4d64-8891-b440d7cda702-image-registry-private-configuration\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.607993 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.607941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03bab594-36e8-4d64-8891-b440d7cda702-registry-certificates\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.607993 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.607976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-bound-sa-token\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.608064 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.608001 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-registry-tls\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.633522 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.633473 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:28.638468 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.638450 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:28.644034 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.644015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:28.679275 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.679254 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:28.691495 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.691470 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" Jul 15 19:15:28.708799 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.708779 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-image-registry-private-configuration\") pod \"5247a32b-bd1b-4c81-990f-5be7568c79d6\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " Jul 15 19:15:28.708904 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.708812 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5247a32b-bd1b-4c81-990f-5be7568c79d6-ca-trust-extracted\") pod \"5247a32b-bd1b-4c81-990f-5be7568c79d6\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " Jul 15 19:15:28.708904 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.708853 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-certificates\") pod \"5247a32b-bd1b-4c81-990f-5be7568c79d6\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " Jul 15 19:15:28.708904 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.708880 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrx5d\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-kube-api-access-nrx5d\") pod \"5247a32b-bd1b-4c81-990f-5be7568c79d6\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " Jul 15 19:15:28.709069 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.708911 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-trusted-ca\") pod \"5247a32b-bd1b-4c81-990f-5be7568c79d6\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " Jul 15 19:15:28.709069 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.708950 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-bound-sa-token\") pod \"5247a32b-bd1b-4c81-990f-5be7568c79d6\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " Jul 15 19:15:28.709069 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709015 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-installation-pull-secrets\") pod \"5247a32b-bd1b-4c81-990f-5be7568c79d6\" (UID: \"5247a32b-bd1b-4c81-990f-5be7568c79d6\") " Jul 15 19:15:28.709218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03bab594-36e8-4d64-8891-b440d7cda702-ca-trust-extracted\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709143 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03bab594-36e8-4d64-8891-b440d7cda702-installation-pull-secrets\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03bab594-36e8-4d64-8891-b440d7cda702-image-registry-private-configuration\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709380 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03bab594-36e8-4d64-8891-b440d7cda702-registry-certificates\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709380 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709253 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-bound-sa-token\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709380 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-registry-tls\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709380 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03bab594-36e8-4d64-8891-b440d7cda702-trusted-ca\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709585 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwzz9\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-kube-api-access-gwzz9\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.709585 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709423 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5247a32b-bd1b-4c81-990f-5be7568c79d6" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:15:28.710177 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.709873 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5247a32b-bd1b-4c81-990f-5be7568c79d6" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:15:28.710447 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.710416 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5247a32b-bd1b-4c81-990f-5be7568c79d6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5247a32b-bd1b-4c81-990f-5be7568c79d6" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:15:28.711689 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.711521 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03bab594-36e8-4d64-8891-b440d7cda702-ca-trust-extracted\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.711689 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.711614 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5247a32b-bd1b-4c81-990f-5be7568c79d6" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:15:28.712062 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.712035 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03bab594-36e8-4d64-8891-b440d7cda702-trusted-ca\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.712700 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.712236 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5247a32b-bd1b-4c81-990f-5be7568c79d6" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:15:28.712700 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.712500 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5247a32b-bd1b-4c81-990f-5be7568c79d6" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:15:28.712700 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.712580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03bab594-36e8-4d64-8891-b440d7cda702-registry-certificates\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.712700 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.712663 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03bab594-36e8-4d64-8891-b440d7cda702-image-registry-private-configuration\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.712970 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.712946 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03bab594-36e8-4d64-8891-b440d7cda702-installation-pull-secrets\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.713202 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.713179 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-kube-api-access-nrx5d" (OuterVolumeSpecName: "kube-api-access-nrx5d") pod "5247a32b-bd1b-4c81-990f-5be7568c79d6" (UID: "5247a32b-bd1b-4c81-990f-5be7568c79d6"). InnerVolumeSpecName "kube-api-access-nrx5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:15:28.713809 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.713788 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-registry-tls\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.719107 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.719081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwzz9\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-kube-api-access-gwzz9\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.719464 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.719421 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03bab594-36e8-4d64-8891-b440d7cda702-bound-sa-token\") pod \"image-registry-5f6db4b9f-ktvp5\" (UID: \"03bab594-36e8-4d64-8891-b440d7cda702\") " pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.769623 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.769560 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:28.810655 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.810632 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-installation-pull-secrets\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:28.810655 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.810656 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5247a32b-bd1b-4c81-990f-5be7568c79d6-image-registry-private-configuration\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:28.810825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.810668 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5247a32b-bd1b-4c81-990f-5be7568c79d6-ca-trust-extracted\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:28.810825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.810680 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-certificates\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:28.810825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.810692 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrx5d\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-kube-api-access-nrx5d\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:28.810825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.810704 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5247a32b-bd1b-4c81-990f-5be7568c79d6-trusted-ca\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:28.810825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.810716 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-bound-sa-token\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:28.822088 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.822053 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp"] Jul 15 19:15:28.824562 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:28.824534 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod073cd7ca_8452_4583_ba05_99ecec719f03.slice/crio-7895fbe6f3c40a79c7d01ac9f6fb684e6dea24e097ce1391cb70becdaa81d075 WatchSource:0}: Error finding container 7895fbe6f3c40a79c7d01ac9f6fb684e6dea24e097ce1391cb70becdaa81d075: Status 404 returned error can't find the container with id 7895fbe6f3c40a79c7d01ac9f6fb684e6dea24e097ce1391cb70becdaa81d075 Jul 15 19:15:28.839305 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.839125 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7"] Jul 15 19:15:28.841742 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:28.841716 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda07f72be_af31_43dc_9856_0b2e8c3bba48.slice/crio-81997ee58aaa9284a0888078d84beccb6d6f75a437a996d263852bc618da886e WatchSource:0}: Error finding container 81997ee58aaa9284a0888078d84beccb6d6f75a437a996d263852bc618da886e: Status 404 returned error can't find the container with id 81997ee58aaa9284a0888078d84beccb6d6f75a437a996d263852bc618da886e Jul 15 19:15:28.843874 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.843809 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xd8s9\"" Jul 15 19:15:28.851604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.851586 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:28.871744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.871715 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:28.974862 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:28.974839 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6db4b9f-ktvp5"] Jul 15 19:15:28.977281 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:28.977258 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bab594_36e8_4d64_8891_b440d7cda702.slice/crio-d7ab15785ea78b7a004f5a0ff5730f9f97447ff1b2b5977ce9dd685909addf9d WatchSource:0}: Error finding container d7ab15785ea78b7a004f5a0ff5730f9f97447ff1b2b5977ce9dd685909addf9d: Status 404 returned error can't find the container with id d7ab15785ea78b7a004f5a0ff5730f9f97447ff1b2b5977ce9dd685909addf9d Jul 15 19:15:29.637940 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.637905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:29.639580 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.639552 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" event={"ID":"03bab594-36e8-4d64-8891-b440d7cda702","Type":"ContainerStarted","Data":"7242e9b6dd7be3fe581de5fb7094c72bcc70650b5f8839c2d424f2f7a8f29654"} Jul 15 19:15:29.639717 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.639608 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" event={"ID":"03bab594-36e8-4d64-8891-b440d7cda702","Type":"ContainerStarted","Data":"d7ab15785ea78b7a004f5a0ff5730f9f97447ff1b2b5977ce9dd685909addf9d"} Jul 15 19:15:29.640276 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.640256 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:29.642528 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.642430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" event={"ID":"073cd7ca-8452-4583-ba05-99ecec719f03","Type":"ContainerStarted","Data":"7895fbe6f3c40a79c7d01ac9f6fb684e6dea24e097ce1391cb70becdaa81d075"} Jul 15 19:15:29.644373 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.644328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79b98d9579-gjx2j" Jul 15 19:15:29.646441 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.646416 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" event={"ID":"a07f72be-af31-43dc-9856-0b2e8c3bba48","Type":"ContainerStarted","Data":"81997ee58aaa9284a0888078d84beccb6d6f75a437a996d263852bc618da886e"} Jul 15 19:15:29.686202 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.686134 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" podStartSLOduration=1.686096417 podStartE2EDuration="1.686096417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:29.664979036 +0000 UTC m=+63.855856842" watchObservedRunningTime="2026-07-15 19:15:29.686096417 +0000 UTC m=+63.876974222" Jul 15 19:15:29.695724 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.695661 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79b98d9579-gjx2j"] Jul 15 19:15:29.696368 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.696316 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:29.700025 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.700005 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79b98d9579-gjx2j"] Jul 15 19:15:29.819070 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:29.818990 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5247a32b-bd1b-4c81-990f-5be7568c79d6-registry-tls\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:15:30.237521 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:30.237489 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5247a32b-bd1b-4c81-990f-5be7568c79d6" path="/var/lib/kubelet/pods/5247a32b-bd1b-4c81-990f-5be7568c79d6/volumes" Jul 15 19:15:31.937522 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:31.937480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:15:31.940248 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:31.940223 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 19:15:31.950617 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:31.950593 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6aa8fab3-6ea1-41fd-9ed6-182447618470-original-pull-secret\") pod \"global-pull-secret-syncer-64nd4\" (UID: \"6aa8fab3-6ea1-41fd-9ed6-182447618470\") " pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:15:32.038389 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.038334 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:15:32.040958 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.040937 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 19:15:32.051599 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.051576 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9984863-de86-49c4-b3bc-352f5eb74188-metrics-certs\") pod \"network-metrics-daemon-7jgjx\" (UID: \"b9984863-de86-49c4-b3bc-352f5eb74188\") " pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:15:32.137066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.137036 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-64nd4" Jul 15 19:15:32.139453 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.139423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:15:32.142223 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.142200 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 19:15:32.152077 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.152054 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 19:15:32.162555 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.162533 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86k6n\" (UniqueName: \"kubernetes.io/projected/60ca1071-d2d6-496c-961e-23291347bd7a-kube-api-access-86k6n\") pod \"network-check-target-2rf4z\" (UID: \"60ca1071-d2d6-496c-961e-23291347bd7a\") " pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:15:32.238584 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.238566 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pjk9x\"" Jul 15 19:15:32.245049 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.245033 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sk5rd\"" Jul 15 19:15:32.246940 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.246925 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:15:32.253677 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.253658 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7jgjx" Jul 15 19:15:32.954928 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.954880 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-64nd4"] Jul 15 19:15:32.961130 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:32.961102 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa8fab3_6ea1_41fd_9ed6_182447618470.slice/crio-dd7682d93585c862bfbe818646efdf8d276a1fdabcc8a19aafe2ae66e7e25eab WatchSource:0}: Error finding container dd7682d93585c862bfbe818646efdf8d276a1fdabcc8a19aafe2ae66e7e25eab: Status 404 returned error can't find the container with id dd7682d93585c862bfbe818646efdf8d276a1fdabcc8a19aafe2ae66e7e25eab Jul 15 19:15:32.968146 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.968081 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2rf4z"] Jul 15 19:15:32.975315 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:32.975259 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ca1071_d2d6_496c_961e_23291347bd7a.slice/crio-ce2af0e2ff262cc2dcd7bc6f2c1e00756baeebd44c07068b52eb93c69ef86449 WatchSource:0}: Error finding container ce2af0e2ff262cc2dcd7bc6f2c1e00756baeebd44c07068b52eb93c69ef86449: Status 404 returned error can't find the container with id ce2af0e2ff262cc2dcd7bc6f2c1e00756baeebd44c07068b52eb93c69ef86449 Jul 15 19:15:32.987457 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:32.987433 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7jgjx"] Jul 15 19:15:32.990139 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:32.990117 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9984863_de86_49c4_b3bc_352f5eb74188.slice/crio-69ae452f87e1912ce4e1d047bc2e3ccd1bf2f1da9a947729bb0a27e9f445494c WatchSource:0}: Error finding container 69ae452f87e1912ce4e1d047bc2e3ccd1bf2f1da9a947729bb0a27e9f445494c: Status 404 returned error can't find the container with id 69ae452f87e1912ce4e1d047bc2e3ccd1bf2f1da9a947729bb0a27e9f445494c Jul 15 19:15:33.147769 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.147702 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:33.149884 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.149865 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9e5338b-132f-4529-9a5d-8d38e2671420-metrics-tls\") pod \"dns-default-2rggc\" (UID: \"f9e5338b-132f-4529-9a5d-8d38e2671420\") " pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:33.327425 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.327399 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l2njj\"" Jul 15 19:15:33.335335 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.335319 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:33.461273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.461243 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2rggc"] Jul 15 19:15:33.465372 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:33.465314 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e5338b_132f_4529_9a5d_8d38e2671420.slice/crio-76d04e053ae333a1ea304cc2bf3e341f62711cc871068f3e781abd3ea6a47a93 WatchSource:0}: Error finding container 76d04e053ae333a1ea304cc2bf3e341f62711cc871068f3e781abd3ea6a47a93: Status 404 returned error can't find the container with id 76d04e053ae333a1ea304cc2bf3e341f62711cc871068f3e781abd3ea6a47a93 Jul 15 19:15:33.659029 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.658946 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7jgjx" event={"ID":"b9984863-de86-49c4-b3bc-352f5eb74188","Type":"ContainerStarted","Data":"69ae452f87e1912ce4e1d047bc2e3ccd1bf2f1da9a947729bb0a27e9f445494c"} Jul 15 19:15:33.661558 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.661490 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" event={"ID":"073cd7ca-8452-4583-ba05-99ecec719f03","Type":"ContainerStarted","Data":"f8e8157b386e5ebc64e735fe33344afc3907d6bc512ba8831a4acfa8fa3b824c"} Jul 15 19:15:33.662151 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.662103 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:33.663765 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.663719 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" Jul 15 19:15:33.664199 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.664138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-64nd4" event={"ID":"6aa8fab3-6ea1-41fd-9ed6-182447618470","Type":"ContainerStarted","Data":"dd7682d93585c862bfbe818646efdf8d276a1fdabcc8a19aafe2ae66e7e25eab"} Jul 15 19:15:33.665909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.665885 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rggc" event={"ID":"f9e5338b-132f-4529-9a5d-8d38e2671420","Type":"ContainerStarted","Data":"76d04e053ae333a1ea304cc2bf3e341f62711cc871068f3e781abd3ea6a47a93"} Jul 15 19:15:33.668027 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.667690 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" event={"ID":"a07f72be-af31-43dc-9856-0b2e8c3bba48","Type":"ContainerStarted","Data":"eab997c6482e158dd3a9ee0a256ed4b5ee741b9cef79d2eed9f7be7f51d461d5"} Jul 15 19:15:33.669215 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.669192 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2rf4z" event={"ID":"60ca1071-d2d6-496c-961e-23291347bd7a","Type":"ContainerStarted","Data":"ce2af0e2ff262cc2dcd7bc6f2c1e00756baeebd44c07068b52eb93c69ef86449"} Jul 15 19:15:33.680607 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.680278 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-598d8687fc-2gncp" podStartSLOduration=1.690337465 podStartE2EDuration="5.680258691s" totalImagesPullingTime="3.989921226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:33.678876892 +0000 UTC m=+67.869754701" watchObservedRunningTime="2026-07-15 19:15:33.680258691 +0000 UTC m=+67.871136499" Jul 15 19:15:33.713969 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:33.713921 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-657994bc88-52lt7" podStartSLOduration=1.759958282 podStartE2EDuration="5.713903337s" totalImagesPullingTime="3.953945055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:33.713160763 +0000 UTC m=+67.904038569" watchObservedRunningTime="2026-07-15 19:15:33.713903337 +0000 UTC m=+67.904781144" Jul 15 19:15:34.677755 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:34.677322 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7jgjx" event={"ID":"b9984863-de86-49c4-b3bc-352f5eb74188","Type":"ContainerStarted","Data":"4f4e140f36f9d8355ce8984099ab0b35af3b2cc87848cd9e6b2c696fb5216fee"} Jul 15 19:15:35.110668 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.110589 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:35.210972 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.210921 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:35.399902 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.399808 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:35.683459 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.683373 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7jgjx" event={"ID":"b9984863-de86-49c4-b3bc-352f5eb74188","Type":"ContainerStarted","Data":"c546deb207e140cbeec7b71cd7e638242b0c521b047e0404c966304d13298296"} Jul 15 19:15:35.700130 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.700071 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7jgjx" podStartSLOduration=68.221294123 podStartE2EDuration="1m9.700051275s" totalImagesPullingTime="1.478757152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:35.699143624 +0000 UTC m=+69.890021462" watchObservedRunningTime="2026-07-15 19:15:35.700051275 +0000 UTC m=+69.890929081" Jul 15 19:15:35.825737 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.825703 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:35.924903 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.924872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:35.980357 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:35.980265 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:37.483983 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:37.483944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:37.486748 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:37.486722 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9db371c-b7b8-448b-b63c-3be20f40bb63-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-rp5lk\" (UID: \"a9db371c-b7b8-448b-b63c-3be20f40bb63\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:37.736191 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:37.736164 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-jmrwq\"" Jul 15 19:15:37.743414 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:37.743397 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" Jul 15 19:15:38.470967 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.470914 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-rp5lk"] Jul 15 19:15:38.475531 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:38.475503 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9db371c_b7b8_448b_b63c_3be20f40bb63.slice/crio-36d84d59ee05196ad542ee7223cb2661d177a9ec297e1024a7d7709bfbc03f69 WatchSource:0}: Error finding container 36d84d59ee05196ad542ee7223cb2661d177a9ec297e1024a7d7709bfbc03f69: Status 404 returned error can't find the container with id 36d84d59ee05196ad542ee7223cb2661d177a9ec297e1024a7d7709bfbc03f69 Jul 15 19:15:38.695780 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.695698 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2rf4z" event={"ID":"60ca1071-d2d6-496c-961e-23291347bd7a","Type":"ContainerStarted","Data":"239c3c599a7b95c910150b885cda30873d251ea8e47040d103a3df82f485152b"} Jul 15 19:15:38.695780 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.695771 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:15:38.697630 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.697602 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-64nd4" event={"ID":"6aa8fab3-6ea1-41fd-9ed6-182447618470","Type":"ContainerStarted","Data":"71d9698ba024cdec1f3e18a2a804c38a15432d70288a74d9f91ea24ff0005dfb"} Jul 15 19:15:38.698709 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.698686 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" event={"ID":"a9db371c-b7b8-448b-b63c-3be20f40bb63","Type":"ContainerStarted","Data":"36d84d59ee05196ad542ee7223cb2661d177a9ec297e1024a7d7709bfbc03f69"} Jul 15 19:15:38.700224 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.700202 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rggc" event={"ID":"f9e5338b-132f-4529-9a5d-8d38e2671420","Type":"ContainerStarted","Data":"5308fd34d98f59f203a95997d16ff4ae12c831c80e1bf5701df2ee090b12e85e"} Jul 15 19:15:38.700340 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.700239 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2rggc" event={"ID":"f9e5338b-132f-4529-9a5d-8d38e2671420","Type":"ContainerStarted","Data":"fee0cae4004ef2af2cace9b573e55a16e4fac907c6be276aea62225f36419de4"} Jul 15 19:15:38.700340 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.700329 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:38.715446 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.715412 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2rf4z" podStartSLOduration=67.374985071 podStartE2EDuration="1m12.715394718s" totalImagesPullingTime="5.340409647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:38.714849337 +0000 UTC m=+72.905727141" watchObservedRunningTime="2026-07-15 19:15:38.715394718 +0000 UTC m=+72.906272576" Jul 15 19:15:38.736236 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.736202 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-64nd4" podStartSLOduration=67.374958488 podStartE2EDuration="1m12.736192857s" totalImagesPullingTime="5.361234369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:38.735582006 +0000 UTC m=+72.926459832" watchObservedRunningTime="2026-07-15 19:15:38.736192857 +0000 UTC m=+72.927070661" Jul 15 19:15:38.756506 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:38.756473 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2rggc" podStartSLOduration=32.912075444 podStartE2EDuration="37.756460847s" totalImagesPullingTime="4.844385403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:38.755936862 +0000 UTC m=+72.946814693" watchObservedRunningTime="2026-07-15 19:15:38.756460847 +0000 UTC m=+72.947338651" Jul 15 19:15:39.109602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.109570 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:39.603902 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.603864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:39.606322 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.606298 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1759d7c-a121-4bc4-9d93-7c9e62deb83b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xhtm6\" (UID: \"c1759d7c-a121-4bc4-9d93-7c9e62deb83b\") " pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:39.791585 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.791565 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gg6xc\"" Jul 15 19:15:39.798753 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.798737 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xhtm6" Jul 15 19:15:39.893917 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.893890 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:39.930714 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.930687 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xhtm6"] Jul 15 19:15:39.932620 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:39.932598 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1759d7c_a121_4bc4_9d93_7c9e62deb83b.slice/crio-f28e7e0402859ee87bdeb348a6370f7b7de1764b05c909b2f2db0a3f66987c58 WatchSource:0}: Error finding container f28e7e0402859ee87bdeb348a6370f7b7de1764b05c909b2f2db0a3f66987c58: Status 404 returned error can't find the container with id f28e7e0402859ee87bdeb348a6370f7b7de1764b05c909b2f2db0a3f66987c58 Jul 15 19:15:39.994525 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:39.994483 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:40.711575 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:40.711544 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" event={"ID":"a9db371c-b7b8-448b-b63c-3be20f40bb63","Type":"ContainerStarted","Data":"7056d9545efd7f066b458ef9e7e07acb5d7c5a229d232eb9a1a835718d938612"} Jul 15 19:15:40.712607 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:40.712583 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhtm6" event={"ID":"c1759d7c-a121-4bc4-9d93-7c9e62deb83b","Type":"ContainerStarted","Data":"f28e7e0402859ee87bdeb348a6370f7b7de1764b05c909b2f2db0a3f66987c58"} Jul 15 19:15:40.735680 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:40.735638 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-rp5lk" podStartSLOduration=34.531406736 podStartE2EDuration="35.73562091s" totalImagesPullingTime="1.204214174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:40.734678742 +0000 UTC m=+74.925556572" watchObservedRunningTime="2026-07-15 19:15:40.73562091 +0000 UTC m=+74.926498715" Jul 15 19:15:40.858983 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:40.858948 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:40.921211 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:40.921185 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:41.320708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:41.320671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:41.388728 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:41.388685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:41.750369 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:41.750327 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:45.732036 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:45.731958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhtm6" event={"ID":"c1759d7c-a121-4bc4-9d93-7c9e62deb83b","Type":"ContainerStarted","Data":"03acbba0e7b851b77e50f7d5ed86df95f453d87f4617fae02a9c9c23edda2717"} Jul 15 19:15:46.137673 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:46.137644 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:46.244122 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:46.244081 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:46.339043 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:46.338999 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:46.397284 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:46.397190 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:46.669782 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:46.669698 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:46.773685 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:46.773650 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:46.939701 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:46.939630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.145852 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.145792 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.211273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.211188 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.337252 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.337220 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.400847 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.400817 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.486326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.486290 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.724047 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.723980 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.793901 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.792543 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zvxz9"] Jul 15 19:15:47.807586 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.806647 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.809420 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.809240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 19:15:47.809879 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.809692 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 19:15:47.809879 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.809769 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 19:15:47.811839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.810377 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 19:15:47.811839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.810601 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-znk59\"" Jul 15 19:15:47.811839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.810757 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 19:15:47.811839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.810934 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 19:15:47.850256 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.849846 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.913724 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.913662 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973009 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-textfile\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973051 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973083 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-sys\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973130 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-root\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973154 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqdm\" (UniqueName: \"kubernetes.io/projected/4fd1a80e-868b-497a-968f-7c4c187954cd-kube-api-access-sgqdm\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1a80e-868b-497a-968f-7c4c187954cd-metrics-client-ca\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-tls\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:47.973363 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:47.973285 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-wtmp\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.010747 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.010710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.074604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-root\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074564 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqdm\" (UniqueName: \"kubernetes.io/projected/4fd1a80e-868b-497a-968f-7c4c187954cd-kube-api-access-sgqdm\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1a80e-868b-497a-968f-7c4c187954cd-metrics-client-ca\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-tls\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074682 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-wtmp\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074722 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-root\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-textfile\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.074851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-sys\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.075184 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.074871 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-sys\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.075184 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.075162 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-textfile\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.075318 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.075276 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4fd1a80e-868b-497a-968f-7c4c187954cd-metrics-client-ca\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.075318 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.075293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-wtmp\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.075567 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.075541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.077699 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.077667 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-tls\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.077774 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.077721 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4fd1a80e-868b-497a-968f-7c4c187954cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.088082 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.088031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqdm\" (UniqueName: \"kubernetes.io/projected/4fd1a80e-868b-497a-968f-7c4c187954cd-kube-api-access-sgqdm\") pod \"node-exporter-zvxz9\" (UID: \"4fd1a80e-868b-497a-968f-7c4c187954cd\") " pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.116718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.116654 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.118668 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.118623 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zvxz9" Jul 15 19:15:48.240799 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.240768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.556281 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.556237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.651797 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.651749 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.710064 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.710036 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2rggc" Jul 15 19:15:48.735562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.735531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.797393 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.797361 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:48.857706 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.857623 2568 patch_prober.go:28] interesting pod/image-registry-5f6db4b9f-ktvp5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 19:15:48.857706 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.857685 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" podUID="03bab594-36e8-4d64-8891-b440d7cda702" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 19:15:48.875803 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:48.875762 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:49.093510 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:49.093483 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:49.168888 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:49.168861 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd1a80e_868b_497a_968f_7c4c187954cd.slice/crio-e5b3f1165f68b5bd780b6dc32eec0d49c7179cc7c74971f7beaa8ca32621b442 WatchSource:0}: Error finding container e5b3f1165f68b5bd780b6dc32eec0d49c7179cc7c74971f7beaa8ca32621b442: Status 404 returned error can't find the container with id e5b3f1165f68b5bd780b6dc32eec0d49c7179cc7c74971f7beaa8ca32621b442 Jul 15 19:15:49.196325 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:49.196299 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:49.750551 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:49.750514 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhtm6" event={"ID":"c1759d7c-a121-4bc4-9d93-7c9e62deb83b","Type":"ContainerStarted","Data":"56ae635fa63067da3b8d55af623c844769fbf87067b1854dd44b8ec0546c6ec7"} Jul 15 19:15:49.751846 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:49.751814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvxz9" event={"ID":"4fd1a80e-868b-497a-968f-7c4c187954cd","Type":"ContainerStarted","Data":"e5b3f1165f68b5bd780b6dc32eec0d49c7179cc7c74971f7beaa8ca32621b442"} Jul 15 19:15:50.178802 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:50.178760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:50.751633 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:50.751596 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:50.756880 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:50.756847 2568 generic.go:356] "Generic (PLEG): container finished" podID="4fd1a80e-868b-497a-968f-7c4c187954cd" containerID="3104b313313e37a9fc205d6b02f7c563c65930c035eb7ffe2e98d93c24172ca6" exitCode=0 Jul 15 19:15:50.757031 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:50.756924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvxz9" event={"ID":"4fd1a80e-868b-497a-968f-7c4c187954cd","Type":"ContainerDied","Data":"3104b313313e37a9fc205d6b02f7c563c65930c035eb7ffe2e98d93c24172ca6"} Jul 15 19:15:50.807632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:50.807608 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:50.932029 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:50.932000 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:51.030307 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:51.030221 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:51.654625 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:51.654597 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f6db4b9f-ktvp5" Jul 15 19:15:51.763077 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:51.763034 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvxz9" event={"ID":"4fd1a80e-868b-497a-968f-7c4c187954cd","Type":"ContainerStarted","Data":"9a6bf4e37e413bd5a7f1f61add93198789abac4b1cd7aab87633f26255a826d5"} Jul 15 19:15:51.763251 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:51.763089 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zvxz9" event={"ID":"4fd1a80e-868b-497a-968f-7c4c187954cd","Type":"ContainerStarted","Data":"ceef07fef87eed0061e78c6a2a8042b050353e96c8768ee24bd4463784965ad9"} Jul 15 19:15:51.788050 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:51.788002 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zvxz9" podStartSLOduration=3.82144932 podStartE2EDuration="4.787985882s" totalImagesPullingTime="903.189949ms" totalInitContainerRuntime="63.346613ms" isStatefulPod=true podCreationTimestamp="2026-07-15 19:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:51.786084585 +0000 UTC m=+85.976962411" watchObservedRunningTime="2026-07-15 19:15:51.787985882 +0000 UTC m=+85.978863687" Jul 15 19:15:51.812405 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:51.812374 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:51.911890 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:51.911806 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:52.013736 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:52.013707 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:52.543395 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:52.543364 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:52.648988 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:52.648957 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:53.125222 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:53.125138 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:54.127436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:54.127409 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:54.777119 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:54.777079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xhtm6" event={"ID":"c1759d7c-a121-4bc4-9d93-7c9e62deb83b","Type":"ContainerStarted","Data":"987a3040ae9cb07a5ae14351e75d9b39430d0f9dbb9e54306e9ec84a5e667080"} Jul 15 19:15:54.797039 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:54.796989 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xhtm6" podStartSLOduration=33.838948549 podStartE2EDuration="47.796972326s" totalImagesPullingTime="13.958023777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 19:15:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:15:54.795902545 +0000 UTC m=+88.986780350" watchObservedRunningTime="2026-07-15 19:15:54.796972326 +0000 UTC m=+88.987850130" Jul 15 19:15:55.406629 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.406598 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7567c8bb9b-mxk9p"] Jul 15 19:15:55.409622 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.409605 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.412039 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.412018 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jjjjr\"" Jul 15 19:15:55.412813 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.412795 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 19:15:55.413365 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.413336 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 19:15:55.413444 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.413371 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 19:15:55.413675 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.413658 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 19:15:55.413890 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.413875 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 19:15:55.413942 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.413879 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 19:15:55.414394 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.414376 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 19:15:55.420018 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.419998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7567c8bb9b-mxk9p"] Jul 15 19:15:55.536480 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.536459 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-config\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.536581 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.536491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-serving-cert\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.536581 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.536518 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-oauth-serving-cert\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.536581 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.536563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-oauth-config\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.536695 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.536584 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-service-ca\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.536695 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.536607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x87v\" (UniqueName: \"kubernetes.io/projected/43e0210c-ca94-4bef-a464-32bf2747b3f6-kube-api-access-7x87v\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637193 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-oauth-serving-cert\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637295 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637201 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-oauth-config\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637295 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-service-ca\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637295 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x87v\" (UniqueName: \"kubernetes.io/projected/43e0210c-ca94-4bef-a464-32bf2747b3f6-kube-api-access-7x87v\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637432 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-config\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637470 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637437 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-serving-cert\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637891 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637873 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-service-ca\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.637973 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.637931 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-oauth-serving-cert\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.638078 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.638058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-config\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.639402 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.639384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-oauth-config\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.639585 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.639569 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-serving-cert\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.646181 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.646162 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x87v\" (UniqueName: \"kubernetes.io/projected/43e0210c-ca94-4bef-a464-32bf2747b3f6-kube-api-access-7x87v\") pod \"console-7567c8bb9b-mxk9p\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.718168 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.718121 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:15:55.842801 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:55.842691 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7567c8bb9b-mxk9p"] Jul 15 19:15:55.845125 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:15:55.845100 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e0210c_ca94_4bef_a464_32bf2747b3f6.slice/crio-ad5ce3ddce77a1d4f0809a70a1bddbd8a64d9a335ab6c57c109f0181e7fcd255 WatchSource:0}: Error finding container ad5ce3ddce77a1d4f0809a70a1bddbd8a64d9a335ab6c57c109f0181e7fcd255: Status 404 returned error can't find the container with id ad5ce3ddce77a1d4f0809a70a1bddbd8a64d9a335ab6c57c109f0181e7fcd255 Jul 15 19:15:56.200572 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:56.200544 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:56.314641 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:56.314614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:56.785814 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:56.785785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567c8bb9b-mxk9p" event={"ID":"43e0210c-ca94-4bef-a464-32bf2747b3f6","Type":"ContainerStarted","Data":"ad5ce3ddce77a1d4f0809a70a1bddbd8a64d9a335ab6c57c109f0181e7fcd255"} Jul 15 19:15:58.680483 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:58.680450 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:58.745650 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:58.745610 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:59.102782 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:59.102747 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:15:59.211384 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:15:59.211331 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:02.807289 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:02.807248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567c8bb9b-mxk9p" event={"ID":"43e0210c-ca94-4bef-a464-32bf2747b3f6","Type":"ContainerStarted","Data":"85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620"} Jul 15 19:16:02.829728 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:02.829684 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7567c8bb9b-mxk9p" podStartSLOduration=1.8176805379999998 podStartE2EDuration="7.829668929s" totalImagesPullingTime="6.011988391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:16:02.828317087 +0000 UTC m=+97.019194892" watchObservedRunningTime="2026-07-15 19:16:02.829668929 +0000 UTC m=+97.020546796" Jul 15 19:16:03.813833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:03.813805 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:03.913782 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:03.913760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:05.028590 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.028559 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-df9c77bc8-64tr6"] Jul 15 19:16:05.078093 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.078066 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df9c77bc8-64tr6"] Jul 15 19:16:05.078229 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.078174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.085109 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.085089 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 19:16:05.206921 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.206893 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-console-config\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.207042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.206925 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-oauth-config\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.207042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.206948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-oauth-serving-cert\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.207042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.206991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpscx\" (UniqueName: \"kubernetes.io/projected/456c02be-d410-419e-96d9-845ed18ff07e-kube-api-access-xpscx\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.207144 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.207039 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-service-ca\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.207144 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.207065 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-serving-cert\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.207144 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.207086 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-trusted-ca-bundle\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308191 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308134 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-console-config\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308191 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308163 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-oauth-config\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308191 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-oauth-serving-cert\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308428 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpscx\" (UniqueName: \"kubernetes.io/projected/456c02be-d410-419e-96d9-845ed18ff07e-kube-api-access-xpscx\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308428 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308226 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-service-ca\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308428 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-serving-cert\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308428 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-trusted-ca-bundle\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308879 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-oauth-serving-cert\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.308987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.308966 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-console-config\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.309036 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.309014 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-service-ca\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.309143 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.309128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-trusted-ca-bundle\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.310521 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.310501 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-oauth-config\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.310668 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.310650 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-serving-cert\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.316931 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.316910 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpscx\" (UniqueName: \"kubernetes.io/projected/456c02be-d410-419e-96d9-845ed18ff07e-kube-api-access-xpscx\") pod \"console-df9c77bc8-64tr6\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.387247 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.387222 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:05.541073 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.541050 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-df9c77bc8-64tr6"] Jul 15 19:16:05.543502 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:16:05.543479 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456c02be_d410_419e_96d9_845ed18ff07e.slice/crio-63051502668d1496eaee5427ac7778bbbe3bb67ae614b1a6ee5e9da0aee7e8e4 WatchSource:0}: Error finding container 63051502668d1496eaee5427ac7778bbbe3bb67ae614b1a6ee5e9da0aee7e8e4: Status 404 returned error can't find the container with id 63051502668d1496eaee5427ac7778bbbe3bb67ae614b1a6ee5e9da0aee7e8e4 Jul 15 19:16:05.719147 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.719086 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:16:05.719405 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.719375 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:16:05.724209 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.724190 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:16:05.823010 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.822976 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df9c77bc8-64tr6" event={"ID":"456c02be-d410-419e-96d9-845ed18ff07e","Type":"ContainerStarted","Data":"3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79"} Jul 15 19:16:05.823127 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.823049 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df9c77bc8-64tr6" event={"ID":"456c02be-d410-419e-96d9-845ed18ff07e","Type":"ContainerStarted","Data":"63051502668d1496eaee5427ac7778bbbe3bb67ae614b1a6ee5e9da0aee7e8e4"} Jul 15 19:16:05.826890 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.826870 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:16:05.843225 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:05.843146 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-df9c77bc8-64tr6" podStartSLOduration=0.843133881 podStartE2EDuration="843.133881ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:16:05.84178436 +0000 UTC m=+100.032662184" watchObservedRunningTime="2026-07-15 19:16:05.843133881 +0000 UTC m=+100.034011684" Jul 15 19:16:06.171539 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:06.171460 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:06.230754 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:06.230726 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:07.755727 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:07.755696 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:07.815413 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:07.815386 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:09.532874 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:09.532829 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:09.631614 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:09.631583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:09.709663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:09.709643 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2rf4z" Jul 15 19:16:11.588526 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:11.588498 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:11.650564 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:11.650539 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:13.007685 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:13.007657 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:13.107970 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:13.107942 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:14.276322 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:14.276288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:15.387902 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:15.387870 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:15.388290 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:15.387947 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:15.393889 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:15.393870 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:15.863658 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:15.863631 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:16:15.914671 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:15.914646 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7567c8bb9b-mxk9p"] Jul 15 19:16:17.146014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:17.145980 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:18.601559 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:18.601532 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:18.702657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:18.702624 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:19.177751 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:19.177725 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:24.510778 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:24.510750 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:24.611482 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:24.611456 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:39.513122 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:39.513082 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:39.577529 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:39.577497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:40.934853 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:40.934812 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7567c8bb9b-mxk9p" podUID="43e0210c-ca94-4bef-a464-32bf2747b3f6" containerName="console" containerID="cri-o://85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620" gracePeriod=15 Jul 15 19:16:41.184442 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.184419 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:16:41.263136 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263105 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-config\") pod \"43e0210c-ca94-4bef-a464-32bf2747b3f6\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " Jul 15 19:16:41.263308 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263147 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x87v\" (UniqueName: \"kubernetes.io/projected/43e0210c-ca94-4bef-a464-32bf2747b3f6-kube-api-access-7x87v\") pod \"43e0210c-ca94-4bef-a464-32bf2747b3f6\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " Jul 15 19:16:41.263308 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263187 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-oauth-serving-cert\") pod \"43e0210c-ca94-4bef-a464-32bf2747b3f6\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " Jul 15 19:16:41.263308 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263208 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-serving-cert\") pod \"43e0210c-ca94-4bef-a464-32bf2747b3f6\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " Jul 15 19:16:41.263308 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263231 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-oauth-config\") pod \"43e0210c-ca94-4bef-a464-32bf2747b3f6\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " Jul 15 19:16:41.263308 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263257 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-service-ca\") pod \"43e0210c-ca94-4bef-a464-32bf2747b3f6\" (UID: \"43e0210c-ca94-4bef-a464-32bf2747b3f6\") " Jul 15 19:16:41.263688 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263640 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-config" (OuterVolumeSpecName: "console-config") pod "43e0210c-ca94-4bef-a464-32bf2747b3f6" (UID: "43e0210c-ca94-4bef-a464-32bf2747b3f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:16:41.263688 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263657 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "43e0210c-ca94-4bef-a464-32bf2747b3f6" (UID: "43e0210c-ca94-4bef-a464-32bf2747b3f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:16:41.263871 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.263768 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-service-ca" (OuterVolumeSpecName: "service-ca") pod "43e0210c-ca94-4bef-a464-32bf2747b3f6" (UID: "43e0210c-ca94-4bef-a464-32bf2747b3f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:16:41.265457 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.265434 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "43e0210c-ca94-4bef-a464-32bf2747b3f6" (UID: "43e0210c-ca94-4bef-a464-32bf2747b3f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:16:41.265820 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.265801 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "43e0210c-ca94-4bef-a464-32bf2747b3f6" (UID: "43e0210c-ca94-4bef-a464-32bf2747b3f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:16:41.265884 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.265826 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e0210c-ca94-4bef-a464-32bf2747b3f6-kube-api-access-7x87v" (OuterVolumeSpecName: "kube-api-access-7x87v") pod "43e0210c-ca94-4bef-a464-32bf2747b3f6" (UID: "43e0210c-ca94-4bef-a464-32bf2747b3f6"). InnerVolumeSpecName "kube-api-access-7x87v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:16:41.364566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.364537 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-config\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:16:41.364566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.364561 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x87v\" (UniqueName: \"kubernetes.io/projected/43e0210c-ca94-4bef-a464-32bf2747b3f6-kube-api-access-7x87v\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:16:41.364566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.364574 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-oauth-serving-cert\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:16:41.364786 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.364585 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-serving-cert\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:16:41.364786 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.364597 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43e0210c-ca94-4bef-a464-32bf2747b3f6-console-oauth-config\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:16:41.364786 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.364608 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43e0210c-ca94-4bef-a464-32bf2747b3f6-service-ca\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:16:41.949517 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.949486 2568 generic.go:356] "Generic (PLEG): container finished" podID="43e0210c-ca94-4bef-a464-32bf2747b3f6" containerID="85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620" exitCode=0 Jul 15 19:16:41.949905 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.949530 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567c8bb9b-mxk9p" Jul 15 19:16:41.949905 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.949548 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567c8bb9b-mxk9p" event={"ID":"43e0210c-ca94-4bef-a464-32bf2747b3f6","Type":"ContainerDied","Data":"85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620"} Jul 15 19:16:41.949905 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.949598 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567c8bb9b-mxk9p" event={"ID":"43e0210c-ca94-4bef-a464-32bf2747b3f6","Type":"ContainerDied","Data":"ad5ce3ddce77a1d4f0809a70a1bddbd8a64d9a335ab6c57c109f0181e7fcd255"} Jul 15 19:16:41.949905 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.949631 2568 scope.go:117] "RemoveContainer" containerID="85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620" Jul 15 19:16:41.958249 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.958235 2568 scope.go:117] "RemoveContainer" containerID="85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620" Jul 15 19:16:41.958541 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:16:41.958516 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620\": container with ID starting with 85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620 not found: ID does not exist" containerID="85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620" Jul 15 19:16:41.958613 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.958549 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620"} err="failed to get container status \"85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620\": rpc error: code = NotFound desc = could not find container \"85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620\": container with ID starting with 85af44922c94c07a15899b467357526a72e23d16d56d2e261f9dc67f5b24d620 not found: ID does not exist" Jul 15 19:16:41.972069 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.972042 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7567c8bb9b-mxk9p"] Jul 15 19:16:41.977195 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:41.977174 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7567c8bb9b-mxk9p"] Jul 15 19:16:42.230250 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:42.227993 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43e0210c-ca94-4bef-a464-32bf2747b3f6" path="/var/lib/kubelet/pods/43e0210c-ca94-4bef-a464-32bf2747b3f6/volumes" Jul 15 19:16:46.913878 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:46.913830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:16:51.362938 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:16:51.362904 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:03.945152 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:03.945112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:04.010615 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:04.010543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:15.849144 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:15.849106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:15.902105 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:15.902069 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:24.541107 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:24.541010 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:24.646489 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:24.646455 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:30.686628 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.686588 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b57fcc84-r4mvs"] Jul 15 19:17:30.687052 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.686856 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43e0210c-ca94-4bef-a464-32bf2747b3f6" containerName="console" Jul 15 19:17:30.687052 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.686867 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="43e0210c-ca94-4bef-a464-32bf2747b3f6" containerName="console" Jul 15 19:17:30.687052 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.686947 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="43e0210c-ca94-4bef-a464-32bf2747b3f6" containerName="console" Jul 15 19:17:30.690071 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.690054 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.701556 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.701529 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b57fcc84-r4mvs"] Jul 15 19:17:30.748508 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.748475 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:30.806499 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.806463 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-oauth-config\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.806708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.806519 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-serving-cert\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.806708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.806585 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmsc\" (UniqueName: \"kubernetes.io/projected/8f6df6b4-a11e-4582-840c-132fed12cbf1-kube-api-access-csmsc\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.806708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.806653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-oauth-serving-cert\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.806858 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.806707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-service-ca\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.806858 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.806731 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-config\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.806858 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.806752 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-trusted-ca-bundle\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.808956 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.808929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:30.907826 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.907787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmsc\" (UniqueName: \"kubernetes.io/projected/8f6df6b4-a11e-4582-840c-132fed12cbf1-kube-api-access-csmsc\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.907826 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.907831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-oauth-serving-cert\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908060 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.907863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-service-ca\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908060 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.907888 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-config\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908060 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.907905 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-trusted-ca-bundle\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908060 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.907925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-oauth-config\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908267 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.908142 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-serving-cert\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908784 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.908748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-service-ca\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908784 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.908767 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-oauth-serving-cert\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908784 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.908776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-config\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.908955 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.908870 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-trusted-ca-bundle\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.910513 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.910481 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-oauth-config\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.910655 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.910641 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-serving-cert\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.917174 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.917154 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmsc\" (UniqueName: \"kubernetes.io/projected/8f6df6b4-a11e-4582-840c-132fed12cbf1-kube-api-access-csmsc\") pod \"console-6b57fcc84-r4mvs\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:30.999168 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:30.999137 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:31.129370 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:31.129325 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b57fcc84-r4mvs"] Jul 15 19:17:31.132164 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:17:31.132139 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6df6b4_a11e_4582_840c_132fed12cbf1.slice/crio-19124f926caa0077ea7ffe6cde100399dc6257d4448b74dd30d873041ae2339c WatchSource:0}: Error finding container 19124f926caa0077ea7ffe6cde100399dc6257d4448b74dd30d873041ae2339c: Status 404 returned error can't find the container with id 19124f926caa0077ea7ffe6cde100399dc6257d4448b74dd30d873041ae2339c Jul 15 19:17:32.124176 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:32.124145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57fcc84-r4mvs" event={"ID":"8f6df6b4-a11e-4582-840c-132fed12cbf1","Type":"ContainerStarted","Data":"7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e"} Jul 15 19:17:32.124581 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:32.124192 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57fcc84-r4mvs" event={"ID":"8f6df6b4-a11e-4582-840c-132fed12cbf1","Type":"ContainerStarted","Data":"19124f926caa0077ea7ffe6cde100399dc6257d4448b74dd30d873041ae2339c"} Jul 15 19:17:32.144954 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:32.144897 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b57fcc84-r4mvs" podStartSLOduration=2.144877251 podStartE2EDuration="2.144877251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:17:32.144367943 +0000 UTC m=+186.335245743" watchObservedRunningTime="2026-07-15 19:17:32.144877251 +0000 UTC m=+186.335755057" Jul 15 19:17:41.000125 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.000090 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:41.000631 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.000159 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:41.005763 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.005739 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:41.160456 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.160436 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:17:41.213731 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.213704 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df9c77bc8-64tr6"] Jul 15 19:17:41.328896 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.328830 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:41.386908 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.386882 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:41.460474 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:41.460452 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:44.223503 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:44.223470 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:17:44.332663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:17:44.332629 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:06.238854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.238803 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-df9c77bc8-64tr6" podUID="456c02be-d410-419e-96d9-845ed18ff07e" containerName="console" containerID="cri-o://3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79" gracePeriod=15 Jul 15 19:18:06.477795 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.477771 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:18:06.564171 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564079 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-console-config\") pod \"456c02be-d410-419e-96d9-845ed18ff07e\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " Jul 15 19:18:06.564171 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564117 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpscx\" (UniqueName: \"kubernetes.io/projected/456c02be-d410-419e-96d9-845ed18ff07e-kube-api-access-xpscx\") pod \"456c02be-d410-419e-96d9-845ed18ff07e\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " Jul 15 19:18:06.564171 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564154 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-oauth-config\") pod \"456c02be-d410-419e-96d9-845ed18ff07e\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " Jul 15 19:18:06.564171 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564171 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-serving-cert\") pod \"456c02be-d410-419e-96d9-845ed18ff07e\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " Jul 15 19:18:06.564523 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564208 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-oauth-serving-cert\") pod \"456c02be-d410-419e-96d9-845ed18ff07e\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " Jul 15 19:18:06.564523 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564235 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-trusted-ca-bundle\") pod \"456c02be-d410-419e-96d9-845ed18ff07e\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " Jul 15 19:18:06.564523 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564258 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-service-ca\") pod \"456c02be-d410-419e-96d9-845ed18ff07e\" (UID: \"456c02be-d410-419e-96d9-845ed18ff07e\") " Jul 15 19:18:06.564758 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564691 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-console-config" (OuterVolumeSpecName: "console-config") pod "456c02be-d410-419e-96d9-845ed18ff07e" (UID: "456c02be-d410-419e-96d9-845ed18ff07e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:18:06.564890 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564810 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "456c02be-d410-419e-96d9-845ed18ff07e" (UID: "456c02be-d410-419e-96d9-845ed18ff07e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:18:06.564890 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.564855 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-service-ca" (OuterVolumeSpecName: "service-ca") pod "456c02be-d410-419e-96d9-845ed18ff07e" (UID: "456c02be-d410-419e-96d9-845ed18ff07e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:18:06.565024 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.565001 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "456c02be-d410-419e-96d9-845ed18ff07e" (UID: "456c02be-d410-419e-96d9-845ed18ff07e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:18:06.566374 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.566332 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "456c02be-d410-419e-96d9-845ed18ff07e" (UID: "456c02be-d410-419e-96d9-845ed18ff07e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:18:06.566495 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.566465 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "456c02be-d410-419e-96d9-845ed18ff07e" (UID: "456c02be-d410-419e-96d9-845ed18ff07e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:18:06.566549 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.566511 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/456c02be-d410-419e-96d9-845ed18ff07e-kube-api-access-xpscx" (OuterVolumeSpecName: "kube-api-access-xpscx") pod "456c02be-d410-419e-96d9-845ed18ff07e" (UID: "456c02be-d410-419e-96d9-845ed18ff07e"). InnerVolumeSpecName "kube-api-access-xpscx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:18:06.665234 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.665194 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-oauth-serving-cert\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:06.665234 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.665227 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-trusted-ca-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:06.665234 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.665240 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-service-ca\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:06.665494 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.665251 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/456c02be-d410-419e-96d9-845ed18ff07e-console-config\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:06.665494 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.665262 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpscx\" (UniqueName: \"kubernetes.io/projected/456c02be-d410-419e-96d9-845ed18ff07e-kube-api-access-xpscx\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:06.665494 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.665273 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-oauth-config\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:06.665494 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:06.665284 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/456c02be-d410-419e-96d9-845ed18ff07e-console-serving-cert\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:07.230687 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.230654 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:07.247268 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.247240 2568 generic.go:356] "Generic (PLEG): container finished" podID="456c02be-d410-419e-96d9-845ed18ff07e" containerID="3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79" exitCode=0 Jul 15 19:18:07.247682 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.247278 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df9c77bc8-64tr6" event={"ID":"456c02be-d410-419e-96d9-845ed18ff07e","Type":"ContainerDied","Data":"3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79"} Jul 15 19:18:07.247682 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.247292 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-df9c77bc8-64tr6" Jul 15 19:18:07.247682 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.247311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-df9c77bc8-64tr6" event={"ID":"456c02be-d410-419e-96d9-845ed18ff07e","Type":"ContainerDied","Data":"63051502668d1496eaee5427ac7778bbbe3bb67ae614b1a6ee5e9da0aee7e8e4"} Jul 15 19:18:07.247682 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.247330 2568 scope.go:117] "RemoveContainer" containerID="3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79" Jul 15 19:18:07.255652 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.255633 2568 scope.go:117] "RemoveContainer" containerID="3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79" Jul 15 19:18:07.255898 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:18:07.255881 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79\": container with ID starting with 3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79 not found: ID does not exist" containerID="3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79" Jul 15 19:18:07.255935 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.255909 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79"} err="failed to get container status \"3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79\": rpc error: code = NotFound desc = could not find container \"3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79\": container with ID starting with 3168b1ec8dcd518d6c37b00072a68965bee60487134ffa717979f637fca83d79 not found: ID does not exist" Jul 15 19:18:07.272934 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.272900 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-df9c77bc8-64tr6"] Jul 15 19:18:07.278138 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.278116 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-df9c77bc8-64tr6"] Jul 15 19:18:07.307011 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.306981 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:07.981889 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.981851 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z"] Jul 15 19:18:07.982141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.982129 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="456c02be-d410-419e-96d9-845ed18ff07e" containerName="console" Jul 15 19:18:07.982189 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.982143 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="456c02be-d410-419e-96d9-845ed18ff07e" containerName="console" Jul 15 19:18:07.982189 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.982182 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="456c02be-d410-419e-96d9-845ed18ff07e" containerName="console" Jul 15 19:18:07.986422 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.986405 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:07.988844 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.988822 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:18:07.989569 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.989552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r7jsk\"" Jul 15 19:18:07.989669 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.989556 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:18:07.994613 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:07.994586 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z"] Jul 15 19:18:08.077281 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.077243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.077281 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.077287 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.077534 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.077315 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftpp\" (UniqueName: \"kubernetes.io/projected/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-kube-api-access-xftpp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.177710 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.177678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.177710 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.177711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.177892 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.177732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftpp\" (UniqueName: \"kubernetes.io/projected/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-kube-api-access-xftpp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.178035 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.178017 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.178075 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.178061 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.187027 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.186994 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftpp\" (UniqueName: \"kubernetes.io/projected/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-kube-api-access-xftpp\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.227456 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.227419 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="456c02be-d410-419e-96d9-845ed18ff07e" path="/var/lib/kubelet/pods/456c02be-d410-419e-96d9-845ed18ff07e/volumes" Jul 15 19:18:08.296080 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.296045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:08.425916 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:08.425889 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z"] Jul 15 19:18:08.428136 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:18:08.428111 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8cf18d_55dc_4c9a_aaba_f9c39d777222.slice/crio-ae05a56a3ecc1e15110cecffb38ab6069fa96ed2a820c578af82f561575ef297 WatchSource:0}: Error finding container ae05a56a3ecc1e15110cecffb38ab6069fa96ed2a820c578af82f561575ef297: Status 404 returned error can't find the container with id ae05a56a3ecc1e15110cecffb38ab6069fa96ed2a820c578af82f561575ef297 Jul 15 19:18:09.257464 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:09.257428 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" event={"ID":"3a8cf18d-55dc-4c9a-aaba-f9c39d777222","Type":"ContainerStarted","Data":"ae05a56a3ecc1e15110cecffb38ab6069fa96ed2a820c578af82f561575ef297"} Jul 15 19:18:15.280635 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:15.280599 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerID="ca2009dc87a48539a95a71ef6b23e7477ee01659a8f1445233df8bc34f4b45f7" exitCode=0 Jul 15 19:18:15.281005 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:15.280671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" event={"ID":"3a8cf18d-55dc-4c9a-aaba-f9c39d777222","Type":"ContainerDied","Data":"ca2009dc87a48539a95a71ef6b23e7477ee01659a8f1445233df8bc34f4b45f7"} Jul 15 19:18:18.291335 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:18.291302 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerID="537b54157d84c7d8e18fbc18d014bb5bbf60aec713138b5671396c7a99969216" exitCode=0 Jul 15 19:18:18.291719 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:18.291393 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" event={"ID":"3a8cf18d-55dc-4c9a-aaba-f9c39d777222","Type":"ContainerDied","Data":"537b54157d84c7d8e18fbc18d014bb5bbf60aec713138b5671396c7a99969216"} Jul 15 19:18:28.330512 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:28.330481 2568 generic.go:356] "Generic (PLEG): container finished" podID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerID="358ec302676f258cccd458ee0e9533f2e899291c2c5b0c53947d27ed7f4cf437" exitCode=0 Jul 15 19:18:28.330899 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:28.330567 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" event={"ID":"3a8cf18d-55dc-4c9a-aaba-f9c39d777222","Type":"ContainerDied","Data":"358ec302676f258cccd458ee0e9533f2e899291c2c5b0c53947d27ed7f4cf437"} Jul 15 19:18:29.514481 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.514456 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:29.646779 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.646716 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-util\") pod \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " Jul 15 19:18:29.646899 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.646786 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xftpp\" (UniqueName: \"kubernetes.io/projected/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-kube-api-access-xftpp\") pod \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " Jul 15 19:18:29.646899 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.646806 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-bundle\") pod \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\" (UID: \"3a8cf18d-55dc-4c9a-aaba-f9c39d777222\") " Jul 15 19:18:29.647501 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.647473 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-bundle" (OuterVolumeSpecName: "bundle") pod "3a8cf18d-55dc-4c9a-aaba-f9c39d777222" (UID: "3a8cf18d-55dc-4c9a-aaba-f9c39d777222"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:18:29.648906 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.648884 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-kube-api-access-xftpp" (OuterVolumeSpecName: "kube-api-access-xftpp") pod "3a8cf18d-55dc-4c9a-aaba-f9c39d777222" (UID: "3a8cf18d-55dc-4c9a-aaba-f9c39d777222"). InnerVolumeSpecName "kube-api-access-xftpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:18:29.651862 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.651841 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-util" (OuterVolumeSpecName: "util") pod "3a8cf18d-55dc-4c9a-aaba-f9c39d777222" (UID: "3a8cf18d-55dc-4c9a-aaba-f9c39d777222"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:18:29.748049 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.748028 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftpp\" (UniqueName: \"kubernetes.io/projected/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-kube-api-access-xftpp\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:29.748049 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.748050 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:29.748169 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:29.748059 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a8cf18d-55dc-4c9a-aaba-f9c39d777222-util\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:30.338219 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:30.338191 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" event={"ID":"3a8cf18d-55dc-4c9a-aaba-f9c39d777222","Type":"ContainerDied","Data":"ae05a56a3ecc1e15110cecffb38ab6069fa96ed2a820c578af82f561575ef297"} Jul 15 19:18:30.338367 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:30.338237 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae05a56a3ecc1e15110cecffb38ab6069fa96ed2a820c578af82f561575ef297" Jul 15 19:18:30.338367 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:30.338203 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540l574z" Jul 15 19:18:34.268180 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:34.268143 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:35.006056 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.006021 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:35.958639 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958608 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb"] Jul 15 19:18:35.959014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958854 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerName="util" Jul 15 19:18:35.959014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958865 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerName="util" Jul 15 19:18:35.959014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958876 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerName="extract" Jul 15 19:18:35.959014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958881 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerName="extract" Jul 15 19:18:35.959014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958893 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerName="pull" Jul 15 19:18:35.959014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958899 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerName="pull" Jul 15 19:18:35.959014 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.958947 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a8cf18d-55dc-4c9a-aaba-f9c39d777222" containerName="extract" Jul 15 19:18:35.961966 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.961946 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:35.964392 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.964371 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 15 19:18:35.964508 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.964496 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jqp8r\"" Jul 15 19:18:35.964573 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.964558 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:18:35.980467 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:35.980444 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb"] Jul 15 19:18:36.088880 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.088850 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7eec418-113f-407b-a3b7-207986bd65a9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tkdhb\" (UID: \"f7eec418-113f-407b-a3b7-207986bd65a9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:36.088983 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.088905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwjq\" (UniqueName: \"kubernetes.io/projected/f7eec418-113f-407b-a3b7-207986bd65a9-kube-api-access-rnwjq\") pod \"cert-manager-operator-controller-manager-55c444bf86-tkdhb\" (UID: \"f7eec418-113f-407b-a3b7-207986bd65a9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:36.190169 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.190145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwjq\" (UniqueName: \"kubernetes.io/projected/f7eec418-113f-407b-a3b7-207986bd65a9-kube-api-access-rnwjq\") pod \"cert-manager-operator-controller-manager-55c444bf86-tkdhb\" (UID: \"f7eec418-113f-407b-a3b7-207986bd65a9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:36.190267 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.190197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7eec418-113f-407b-a3b7-207986bd65a9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tkdhb\" (UID: \"f7eec418-113f-407b-a3b7-207986bd65a9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:36.190571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.190553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7eec418-113f-407b-a3b7-207986bd65a9-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tkdhb\" (UID: \"f7eec418-113f-407b-a3b7-207986bd65a9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:36.199300 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.199281 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwjq\" (UniqueName: \"kubernetes.io/projected/f7eec418-113f-407b-a3b7-207986bd65a9-kube-api-access-rnwjq\") pod \"cert-manager-operator-controller-manager-55c444bf86-tkdhb\" (UID: \"f7eec418-113f-407b-a3b7-207986bd65a9\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:36.270847 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.270830 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" Jul 15 19:18:36.397878 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:36.397846 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb"] Jul 15 19:18:36.401110 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:18:36.401087 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eec418_113f_407b_a3b7_207986bd65a9.slice/crio-b2e458c403b1feb11e2496ef6fb2e8abadab99ca7d2e6d4c493f56a4e244cf77 WatchSource:0}: Error finding container b2e458c403b1feb11e2496ef6fb2e8abadab99ca7d2e6d4c493f56a4e244cf77: Status 404 returned error can't find the container with id b2e458c403b1feb11e2496ef6fb2e8abadab99ca7d2e6d4c493f56a4e244cf77 Jul 15 19:18:37.364950 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:37.364917 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" event={"ID":"f7eec418-113f-407b-a3b7-207986bd65a9","Type":"ContainerStarted","Data":"b2e458c403b1feb11e2496ef6fb2e8abadab99ca7d2e6d4c493f56a4e244cf77"} Jul 15 19:18:37.618856 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:37.618777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:37.719510 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:37.719482 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:41.381165 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:41.381133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" event={"ID":"f7eec418-113f-407b-a3b7-207986bd65a9","Type":"ContainerStarted","Data":"f4241fc8ec44967e9b9cfc13cd5ff22e1b14b53b2521cc61338419045d52b62f"} Jul 15 19:18:41.404808 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:41.404702 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tkdhb" podStartSLOduration=1.669830085 podStartE2EDuration="6.404684738s" totalImagesPullingTime="4.734854653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:18:41.402925681 +0000 UTC m=+255.593803485" watchObservedRunningTime="2026-07-15 19:18:41.404684738 +0000 UTC m=+255.595562543" Jul 15 19:18:41.900426 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:41.900392 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:42.000732 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.000704 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:42.133933 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.133903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:42.227446 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.227384 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:42.856151 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.856118 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n"] Jul 15 19:18:42.859608 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.859592 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:42.861960 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.861935 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:18:42.861960 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.861946 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:18:42.862143 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.861947 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r7jsk\"" Jul 15 19:18:42.868732 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.868711 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n"] Jul 15 19:18:42.939490 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.939464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpb98\" (UniqueName: \"kubernetes.io/projected/4047c42a-9073-42f5-ba60-4fe302cb5c48-kube-api-access-dpb98\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:42.939596 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.939509 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:42.939641 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:42.939612 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.040101 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.040078 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.040207 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.040115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpb98\" (UniqueName: \"kubernetes.io/projected/4047c42a-9073-42f5-ba60-4fe302cb5c48-kube-api-access-dpb98\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.040207 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.040144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.040500 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.040480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.040571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.040489 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.049057 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.049037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpb98\" (UniqueName: \"kubernetes.io/projected/4047c42a-9073-42f5-ba60-4fe302cb5c48-kube-api-access-dpb98\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.168902 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.168853 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:43.213838 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.213771 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:43.291402 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.291276 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n"] Jul 15 19:18:43.293722 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:18:43.293697 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4047c42a_9073_42f5_ba60_4fe302cb5c48.slice/crio-fde4f02c1a36995d420b61112d8ac44f8be526baa16f1f62fc2cab0d25d3746e WatchSource:0}: Error finding container fde4f02c1a36995d420b61112d8ac44f8be526baa16f1f62fc2cab0d25d3746e: Status 404 returned error can't find the container with id fde4f02c1a36995d420b61112d8ac44f8be526baa16f1f62fc2cab0d25d3746e Jul 15 19:18:43.314477 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.314441 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:43.390074 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.390036 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" event={"ID":"4047c42a-9073-42f5-ba60-4fe302cb5c48","Type":"ContainerStarted","Data":"0a40b6c2630af370b8d008eccd411d1866195da4dc85b6ec289adc5872a808be"} Jul 15 19:18:43.390074 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:43.390079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" event={"ID":"4047c42a-9073-42f5-ba60-4fe302cb5c48","Type":"ContainerStarted","Data":"fde4f02c1a36995d420b61112d8ac44f8be526baa16f1f62fc2cab0d25d3746e"} Jul 15 19:18:44.395542 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:44.395455 2568 generic.go:356] "Generic (PLEG): container finished" podID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerID="0a40b6c2630af370b8d008eccd411d1866195da4dc85b6ec289adc5872a808be" exitCode=0 Jul 15 19:18:44.395542 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:44.395516 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" event={"ID":"4047c42a-9073-42f5-ba60-4fe302cb5c48","Type":"ContainerDied","Data":"0a40b6c2630af370b8d008eccd411d1866195da4dc85b6ec289adc5872a808be"} Jul 15 19:18:44.610696 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:44.610664 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:44.757818 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:44.757788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:46.404891 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:46.404863 2568 generic.go:356] "Generic (PLEG): container finished" podID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerID="33eb0deb6ba684e614560677fb44907d4a44f8ef62a967ed5353ab5ecde6c9bb" exitCode=0 Jul 15 19:18:46.405213 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:46.404923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" event={"ID":"4047c42a-9073-42f5-ba60-4fe302cb5c48","Type":"ContainerDied","Data":"33eb0deb6ba684e614560677fb44907d4a44f8ef62a967ed5353ab5ecde6c9bb"} Jul 15 19:18:47.411062 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:47.411028 2568 generic.go:356] "Generic (PLEG): container finished" podID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerID="3cc4240eb8d9fabb0ecfaa4d2f645b438e6e7d514ecb2e5435e0aee9772a0ad3" exitCode=0 Jul 15 19:18:47.411433 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:47.411114 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" event={"ID":"4047c42a-9073-42f5-ba60-4fe302cb5c48","Type":"ContainerDied","Data":"3cc4240eb8d9fabb0ecfaa4d2f645b438e6e7d514ecb2e5435e0aee9772a0ad3"} Jul 15 19:18:48.557797 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.557775 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:48.676624 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.676565 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-util\") pod \"4047c42a-9073-42f5-ba60-4fe302cb5c48\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " Jul 15 19:18:48.676746 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.676627 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpb98\" (UniqueName: \"kubernetes.io/projected/4047c42a-9073-42f5-ba60-4fe302cb5c48-kube-api-access-dpb98\") pod \"4047c42a-9073-42f5-ba60-4fe302cb5c48\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " Jul 15 19:18:48.676746 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.676666 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-bundle\") pod \"4047c42a-9073-42f5-ba60-4fe302cb5c48\" (UID: \"4047c42a-9073-42f5-ba60-4fe302cb5c48\") " Jul 15 19:18:48.677017 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.676999 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-bundle" (OuterVolumeSpecName: "bundle") pod "4047c42a-9073-42f5-ba60-4fe302cb5c48" (UID: "4047c42a-9073-42f5-ba60-4fe302cb5c48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:18:48.678744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.678713 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4047c42a-9073-42f5-ba60-4fe302cb5c48-kube-api-access-dpb98" (OuterVolumeSpecName: "kube-api-access-dpb98") pod "4047c42a-9073-42f5-ba60-4fe302cb5c48" (UID: "4047c42a-9073-42f5-ba60-4fe302cb5c48"). InnerVolumeSpecName "kube-api-access-dpb98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:18:48.681440 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.681406 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-util" (OuterVolumeSpecName: "util") pod "4047c42a-9073-42f5-ba60-4fe302cb5c48" (UID: "4047c42a-9073-42f5-ba60-4fe302cb5c48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:18:48.777500 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.777479 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:48.777500 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.777500 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4047c42a-9073-42f5-ba60-4fe302cb5c48-util\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:48.777613 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:48.777509 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpb98\" (UniqueName: \"kubernetes.io/projected/4047c42a-9073-42f5-ba60-4fe302cb5c48-kube-api-access-dpb98\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:18:49.421237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.421206 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" event={"ID":"4047c42a-9073-42f5-ba60-4fe302cb5c48","Type":"ContainerDied","Data":"fde4f02c1a36995d420b61112d8ac44f8be526baa16f1f62fc2cab0d25d3746e"} Jul 15 19:18:49.421407 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.421246 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde4f02c1a36995d420b61112d8ac44f8be526baa16f1f62fc2cab0d25d3746e" Jul 15 19:18:49.421407 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.421245 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvqq9n" Jul 15 19:18:49.813488 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813460 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8"] Jul 15 19:18:49.813807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813694 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerName="util" Jul 15 19:18:49.813807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813704 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerName="util" Jul 15 19:18:49.813807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813713 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerName="extract" Jul 15 19:18:49.813807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813718 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerName="extract" Jul 15 19:18:49.813807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813728 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerName="pull" Jul 15 19:18:49.813807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813733 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerName="pull" Jul 15 19:18:49.813807 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.813778 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4047c42a-9073-42f5-ba60-4fe302cb5c48" containerName="extract" Jul 15 19:18:49.817391 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.817357 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:49.820039 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.820020 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 19:18:49.820229 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.820213 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-2frsc\"" Jul 15 19:18:49.820914 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.820897 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 19:18:49.829868 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.829848 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8"] Jul 15 19:18:49.884140 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.884108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65vc2\" (UniqueName: \"kubernetes.io/projected/bb23614a-1630-4c98-b657-f39c7218132b-kube-api-access-65vc2\") pod \"cert-manager-cainjector-559cdd8c87-vz9t8\" (UID: \"bb23614a-1630-4c98-b657-f39c7218132b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:49.884266 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.884158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb23614a-1630-4c98-b657-f39c7218132b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vz9t8\" (UID: \"bb23614a-1630-4c98-b657-f39c7218132b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:49.984853 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.984825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65vc2\" (UniqueName: \"kubernetes.io/projected/bb23614a-1630-4c98-b657-f39c7218132b-kube-api-access-65vc2\") pod \"cert-manager-cainjector-559cdd8c87-vz9t8\" (UID: \"bb23614a-1630-4c98-b657-f39c7218132b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:49.984996 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.984867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb23614a-1630-4c98-b657-f39c7218132b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vz9t8\" (UID: \"bb23614a-1630-4c98-b657-f39c7218132b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:49.994185 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.994163 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bb23614a-1630-4c98-b657-f39c7218132b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vz9t8\" (UID: \"bb23614a-1630-4c98-b657-f39c7218132b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:49.994303 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:49.994172 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65vc2\" (UniqueName: \"kubernetes.io/projected/bb23614a-1630-4c98-b657-f39c7218132b-kube-api-access-65vc2\") pod \"cert-manager-cainjector-559cdd8c87-vz9t8\" (UID: \"bb23614a-1630-4c98-b657-f39c7218132b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:50.126985 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:50.126892 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" Jul 15 19:18:50.254502 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:50.254404 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8"] Jul 15 19:18:50.256824 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:18:50.256794 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb23614a_1630_4c98_b657_f39c7218132b.slice/crio-10cba015c79dc55488840e1082ea6e029e20d27acd7165270685a5dcab883e93 WatchSource:0}: Error finding container 10cba015c79dc55488840e1082ea6e029e20d27acd7165270685a5dcab883e93: Status 404 returned error can't find the container with id 10cba015c79dc55488840e1082ea6e029e20d27acd7165270685a5dcab883e93 Jul 15 19:18:50.426945 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:50.426860 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" event={"ID":"bb23614a-1630-4c98-b657-f39c7218132b","Type":"ContainerStarted","Data":"10cba015c79dc55488840e1082ea6e029e20d27acd7165270685a5dcab883e93"} Jul 15 19:18:51.821265 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:51.821210 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:51.972550 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:51.972511 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:52.149725 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:52.149655 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:53.310382 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.310331 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:53.369496 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.369465 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:53.440968 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.440944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" event={"ID":"bb23614a-1630-4c98-b657-f39c7218132b","Type":"ContainerStarted","Data":"ed2e3d036aaccbaca86771e1c41374f002f8d41b4247c8fdf8a4aa8f96d6c5fc"} Jul 15 19:18:53.461604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.461558 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vz9t8" podStartSLOduration=2.198330226 podStartE2EDuration="4.461544339s" totalImagesPullingTime="2.263214113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:18:53.46048321 +0000 UTC m=+267.651361016" watchObservedRunningTime="2026-07-15 19:18:53.461544339 +0000 UTC m=+267.652422144" Jul 15 19:18:53.576909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.576837 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf"] Jul 15 19:18:53.579932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.579914 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:53.585223 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.585200 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 19:18:53.585381 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.585201 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 19:18:53.585381 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.585279 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wqlmm\"" Jul 15 19:18:53.591545 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.591526 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf"] Jul 15 19:18:53.709582 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.709558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6223cd64-6bce-41e0-8925-0528c7de52f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4lwwf\" (UID: \"6223cd64-6bce-41e0-8925-0528c7de52f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:53.709708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.709630 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrvvh\" (UniqueName: \"kubernetes.io/projected/6223cd64-6bce-41e0-8925-0528c7de52f0-kube-api-access-mrvvh\") pod \"openshift-lws-operator-fd8ccff4c-4lwwf\" (UID: \"6223cd64-6bce-41e0-8925-0528c7de52f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:53.810895 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.810871 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrvvh\" (UniqueName: \"kubernetes.io/projected/6223cd64-6bce-41e0-8925-0528c7de52f0-kube-api-access-mrvvh\") pod \"openshift-lws-operator-fd8ccff4c-4lwwf\" (UID: \"6223cd64-6bce-41e0-8925-0528c7de52f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:53.811037 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.810915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6223cd64-6bce-41e0-8925-0528c7de52f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4lwwf\" (UID: \"6223cd64-6bce-41e0-8925-0528c7de52f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:53.811292 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.811271 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6223cd64-6bce-41e0-8925-0528c7de52f0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4lwwf\" (UID: \"6223cd64-6bce-41e0-8925-0528c7de52f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:53.821180 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.821160 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrvvh\" (UniqueName: \"kubernetes.io/projected/6223cd64-6bce-41e0-8925-0528c7de52f0-kube-api-access-mrvvh\") pod \"openshift-lws-operator-fd8ccff4c-4lwwf\" (UID: \"6223cd64-6bce-41e0-8925-0528c7de52f0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:53.893744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:53.893672 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" Jul 15 19:18:54.018288 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:54.018265 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf"] Jul 15 19:18:54.020604 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:18:54.020575 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6223cd64_6bce_41e0_8925_0528c7de52f0.slice/crio-c259576f7a0a2d13a8c9c0ac2ae2f332fa3e6a26197975bdf95f0e558010ecec WatchSource:0}: Error finding container c259576f7a0a2d13a8c9c0ac2ae2f332fa3e6a26197975bdf95f0e558010ecec: Status 404 returned error can't find the container with id c259576f7a0a2d13a8c9c0ac2ae2f332fa3e6a26197975bdf95f0e558010ecec Jul 15 19:18:54.445586 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:54.445554 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" event={"ID":"6223cd64-6bce-41e0-8925-0528c7de52f0","Type":"ContainerStarted","Data":"c259576f7a0a2d13a8c9c0ac2ae2f332fa3e6a26197975bdf95f0e558010ecec"} Jul 15 19:18:56.454819 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:56.454788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" event={"ID":"6223cd64-6bce-41e0-8925-0528c7de52f0","Type":"ContainerStarted","Data":"c5b1ae168765fa0fbb198ca7f062224f0fb1223ec07bd66db803b831fe5342a1"} Jul 15 19:18:56.476316 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:56.476271 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4lwwf" podStartSLOduration=1.6686124009999999 podStartE2EDuration="3.476234447s" totalImagesPullingTime="1.807622046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:18:56.475328831 +0000 UTC m=+270.666206636" watchObservedRunningTime="2026-07-15 19:18:56.476234447 +0000 UTC m=+270.667112252" Jul 15 19:18:57.292706 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:57.292675 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:57.396855 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:57.396824 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:59.207478 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.207449 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:18:59.309063 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.309035 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts"] Jul 15 19:18:59.370269 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.370245 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts"] Jul 15 19:18:59.370424 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.370336 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.373506 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.373482 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-r7jsk\"" Jul 15 19:18:59.373806 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.373785 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 19:18:59.374437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.374406 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 19:18:59.453133 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.453105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlh6x\" (UniqueName: \"kubernetes.io/projected/79acb927-c939-47f1-96ad-6e6d3629cd33-kube-api-access-mlh6x\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.453241 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.453147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.453241 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.453216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.554216 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.554193 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.554311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.554246 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlh6x\" (UniqueName: \"kubernetes.io/projected/79acb927-c939-47f1-96ad-6e6d3629cd33-kube-api-access-mlh6x\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.554311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.554276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.554562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.554544 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.554610 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.554595 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.564384 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.564339 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlh6x\" (UniqueName: \"kubernetes.io/projected/79acb927-c939-47f1-96ad-6e6d3629cd33-kube-api-access-mlh6x\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.681667 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.681645 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:18:59.820959 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:18:59.820814 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts"] Jul 15 19:18:59.823873 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:18:59.823849 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79acb927_c939_47f1_96ad_6e6d3629cd33.slice/crio-042a898efaf1aa37eecb7f85ff2dd2c4aeb2bd6b4636c8add0444547545497f3 WatchSource:0}: Error finding container 042a898efaf1aa37eecb7f85ff2dd2c4aeb2bd6b4636c8add0444547545497f3: Status 404 returned error can't find the container with id 042a898efaf1aa37eecb7f85ff2dd2c4aeb2bd6b4636c8add0444547545497f3 Jul 15 19:19:00.470754 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:00.470718 2568 generic.go:356] "Generic (PLEG): container finished" podID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerID="da8a7a5bcfd2373382ac9130be3d6e021c8459d33d91de50194051507b6e553e" exitCode=0 Jul 15 19:19:00.471239 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:00.470769 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" event={"ID":"79acb927-c939-47f1-96ad-6e6d3629cd33","Type":"ContainerDied","Data":"da8a7a5bcfd2373382ac9130be3d6e021c8459d33d91de50194051507b6e553e"} Jul 15 19:19:00.471239 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:00.470814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" event={"ID":"79acb927-c939-47f1-96ad-6e6d3629cd33","Type":"ContainerStarted","Data":"042a898efaf1aa37eecb7f85ff2dd2c4aeb2bd6b4636c8add0444547545497f3"} Jul 15 19:19:00.997619 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:00.997590 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:01.098117 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:01.098086 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:02.480738 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:02.480704 2568 generic.go:356] "Generic (PLEG): container finished" podID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerID="042631bd1f97498e3ee807aac0765d38a7d217bd61c53fc8f44a2a3909302025" exitCode=0 Jul 15 19:19:02.481132 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:02.480780 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" event={"ID":"79acb927-c939-47f1-96ad-6e6d3629cd33","Type":"ContainerDied","Data":"042631bd1f97498e3ee807aac0765d38a7d217bd61c53fc8f44a2a3909302025"} Jul 15 19:19:03.486771 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:03.486740 2568 generic.go:356] "Generic (PLEG): container finished" podID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerID="c12591c98eb383cf3f34e176b9300b994b1f26c2c7b6e82d6bcc2223a910bdac" exitCode=0 Jul 15 19:19:03.487140 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:03.486794 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" event={"ID":"79acb927-c939-47f1-96ad-6e6d3629cd33","Type":"ContainerDied","Data":"c12591c98eb383cf3f34e176b9300b994b1f26c2c7b6e82d6bcc2223a910bdac"} Jul 15 19:19:04.613977 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.613954 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:19:04.789804 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.789743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlh6x\" (UniqueName: \"kubernetes.io/projected/79acb927-c939-47f1-96ad-6e6d3629cd33-kube-api-access-mlh6x\") pod \"79acb927-c939-47f1-96ad-6e6d3629cd33\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " Jul 15 19:19:04.789804 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.789801 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-util\") pod \"79acb927-c939-47f1-96ad-6e6d3629cd33\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " Jul 15 19:19:04.789956 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.789824 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-bundle\") pod \"79acb927-c939-47f1-96ad-6e6d3629cd33\" (UID: \"79acb927-c939-47f1-96ad-6e6d3629cd33\") " Jul 15 19:19:04.790632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.790604 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-bundle" (OuterVolumeSpecName: "bundle") pod "79acb927-c939-47f1-96ad-6e6d3629cd33" (UID: "79acb927-c939-47f1-96ad-6e6d3629cd33"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:19:04.791806 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.791774 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79acb927-c939-47f1-96ad-6e6d3629cd33-kube-api-access-mlh6x" (OuterVolumeSpecName: "kube-api-access-mlh6x") pod "79acb927-c939-47f1-96ad-6e6d3629cd33" (UID: "79acb927-c939-47f1-96ad-6e6d3629cd33"). InnerVolumeSpecName "kube-api-access-mlh6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:19:04.795754 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.795712 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-util" (OuterVolumeSpecName: "util") pod "79acb927-c939-47f1-96ad-6e6d3629cd33" (UID: "79acb927-c939-47f1-96ad-6e6d3629cd33"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:19:04.890995 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.890973 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-util\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:19:04.890995 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.890990 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79acb927-c939-47f1-96ad-6e6d3629cd33-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:19:04.891111 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:04.890999 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlh6x\" (UniqueName: \"kubernetes.io/projected/79acb927-c939-47f1-96ad-6e6d3629cd33-kube-api-access-mlh6x\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:19:05.495987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:05.495963 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" Jul 15 19:19:05.496139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:05.495987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmkfts" event={"ID":"79acb927-c939-47f1-96ad-6e6d3629cd33","Type":"ContainerDied","Data":"042a898efaf1aa37eecb7f85ff2dd2c4aeb2bd6b4636c8add0444547545497f3"} Jul 15 19:19:05.496139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:05.496022 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="042a898efaf1aa37eecb7f85ff2dd2c4aeb2bd6b4636c8add0444547545497f3" Jul 15 19:19:15.439529 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:15.439498 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:15.548574 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:15.548543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:16.290918 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.290888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:16.542834 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.542744 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:16.636872 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.636834 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:16.717151 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717125 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd"] Jul 15 19:19:16.717401 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717386 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerName="pull" Jul 15 19:19:16.717455 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717402 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerName="pull" Jul 15 19:19:16.717455 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717409 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerName="extract" Jul 15 19:19:16.717455 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717415 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerName="extract" Jul 15 19:19:16.717455 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717435 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerName="util" Jul 15 19:19:16.717455 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717441 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerName="util" Jul 15 19:19:16.717603 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.717484 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="79acb927-c939-47f1-96ad-6e6d3629cd33" containerName="extract" Jul 15 19:19:16.722399 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.722366 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.725983 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.725962 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 19:19:16.726643 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.726627 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-c6zmb\"" Jul 15 19:19:16.727120 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.727107 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 19:19:16.727239 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.727224 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 19:19:16.728889 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.728874 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 19:19:16.767100 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.767069 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd"] Jul 15 19:19:16.872650 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.872586 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78359574-4e48-4da2-84b4-a3f0269e58a5-tmp\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.872650 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.872617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5j88\" (UniqueName: \"kubernetes.io/projected/78359574-4e48-4da2-84b4-a3f0269e58a5-kube-api-access-x5j88\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.872853 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.872656 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78359574-4e48-4da2-84b4-a3f0269e58a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.872853 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.872732 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78359574-4e48-4da2-84b4-a3f0269e58a5-manifests\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.872853 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.872768 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78359574-4e48-4da2-84b4-a3f0269e58a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.973920 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.973890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78359574-4e48-4da2-84b4-a3f0269e58a5-tmp\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.974042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.973927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5j88\" (UniqueName: \"kubernetes.io/projected/78359574-4e48-4da2-84b4-a3f0269e58a5-kube-api-access-x5j88\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.974042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.973963 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78359574-4e48-4da2-84b4-a3f0269e58a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.974042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.974009 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78359574-4e48-4da2-84b4-a3f0269e58a5-manifests\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.974202 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.974042 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78359574-4e48-4da2-84b4-a3f0269e58a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.974428 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.974409 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78359574-4e48-4da2-84b4-a3f0269e58a5-manifests\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.976149 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.976126 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78359574-4e48-4da2-84b4-a3f0269e58a5-tmp\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.976277 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.976258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78359574-4e48-4da2-84b4-a3f0269e58a5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.976335 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.976264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78359574-4e48-4da2-84b4-a3f0269e58a5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:16.997267 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:16.997244 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5j88\" (UniqueName: \"kubernetes.io/projected/78359574-4e48-4da2-84b4-a3f0269e58a5-kube-api-access-x5j88\") pod \"opendatahub-operator-controller-manager-5d9998b69-r69fd\" (UID: \"78359574-4e48-4da2-84b4-a3f0269e58a5\") " pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:17.026672 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.026652 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:17.032489 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.032471 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:17.176591 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.176506 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:17.214770 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.214740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd"] Jul 15 19:19:17.218504 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:19:17.218481 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78359574_4e48_4da2_84b4_a3f0269e58a5.slice/crio-862f6518b7d5eb48e6307728b1827d857d8e84608273cfa68069523e574f12fa WatchSource:0}: Error finding container 862f6518b7d5eb48e6307728b1827d857d8e84608273cfa68069523e574f12fa: Status 404 returned error can't find the container with id 862f6518b7d5eb48e6307728b1827d857d8e84608273cfa68069523e574f12fa Jul 15 19:19:17.264374 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.264330 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:17.358537 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.358511 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:17.546130 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.546099 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" event={"ID":"78359574-4e48-4da2-84b4-a3f0269e58a5","Type":"ContainerStarted","Data":"862f6518b7d5eb48e6307728b1827d857d8e84608273cfa68069523e574f12fa"} Jul 15 19:19:17.571703 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.571680 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:17.671380 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:17.671338 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:20.561192 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:20.561165 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" event={"ID":"78359574-4e48-4da2-84b4-a3f0269e58a5","Type":"ContainerStarted","Data":"f8c07ae1fd517f5f283f274ce21a7619958693652cfd6984174f6938752ad062"} Jul 15 19:19:21.567531 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:21.567499 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:21.681175 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:21.681149 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:22.571685 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:22.571655 2568 generic.go:356] "Generic (PLEG): container finished" podID="78359574-4e48-4da2-84b4-a3f0269e58a5" containerID="f8c07ae1fd517f5f283f274ce21a7619958693652cfd6984174f6938752ad062" exitCode=0 Jul 15 19:19:22.572156 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:22.571738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" event={"ID":"78359574-4e48-4da2-84b4-a3f0269e58a5","Type":"ContainerDied","Data":"f8c07ae1fd517f5f283f274ce21a7619958693652cfd6984174f6938752ad062"} Jul 15 19:19:24.580671 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:24.580642 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" event={"ID":"78359574-4e48-4da2-84b4-a3f0269e58a5","Type":"ContainerStarted","Data":"9f5ff62d251776d5b165575ee129d6026a0481f08d6803654704e97451cbcd1c"} Jul 15 19:19:24.581094 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:24.580877 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:24.610841 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:24.610793 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" podStartSLOduration=3.261128572 podStartE2EDuration="8.610777997s" totalImagesPullingTime="4.254709104s" totalInitContainerRuntime="1.094940321s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:19:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:19:24.60925526 +0000 UTC m=+298.800133067" watchObservedRunningTime="2026-07-15 19:19:24.610777997 +0000 UTC m=+298.801655802" Jul 15 19:19:26.136899 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:26.136870 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 19:19:28.193829 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:28.193793 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:29.615225 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:29.615193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:31.094578 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:31.094540 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:31.199046 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:31.199020 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:35.586728 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:35.586701 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d9998b69-r69fd" Jul 15 19:19:37.495395 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:37.495333 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:37.601265 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:37.601233 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:38.620703 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:38.620648 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:38.714486 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:38.714453 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:38.814651 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:38.814612 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:39.156658 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:39.156616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:39.245131 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:39.245093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:39.419501 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:39.419426 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:41.013849 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.013819 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:41.315015 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.314932 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c95658bf4-76mh2"] Jul 15 19:19:41.319584 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.319565 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.323607 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.323583 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 19:19:41.323607 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.323603 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4rmzk\"" Jul 15 19:19:41.323779 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.323606 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 19:19:41.323779 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.323583 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 19:19:41.323779 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.323633 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 19:19:41.329897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.329876 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c95658bf4-76mh2"] Jul 15 19:19:41.347278 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.347248 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:41.349305 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.349288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89ecfac-1ba1-4152-93c3-b97f9d27175e-tls-certs\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.349373 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.349327 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89ecfac-1ba1-4152-93c3-b97f9d27175e-tmp\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.349373 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.349360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nzkg\" (UniqueName: \"kubernetes.io/projected/f89ecfac-1ba1-4152-93c3-b97f9d27175e-kube-api-access-2nzkg\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.450680 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.450647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89ecfac-1ba1-4152-93c3-b97f9d27175e-tls-certs\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.450841 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.450690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89ecfac-1ba1-4152-93c3-b97f9d27175e-tmp\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.450841 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.450709 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nzkg\" (UniqueName: \"kubernetes.io/projected/f89ecfac-1ba1-4152-93c3-b97f9d27175e-kube-api-access-2nzkg\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.452978 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.452954 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f89ecfac-1ba1-4152-93c3-b97f9d27175e-tmp\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.453117 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.453081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f89ecfac-1ba1-4152-93c3-b97f9d27175e-tls-certs\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.460258 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.460230 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nzkg\" (UniqueName: \"kubernetes.io/projected/f89ecfac-1ba1-4152-93c3-b97f9d27175e-kube-api-access-2nzkg\") pod \"kube-auth-proxy-c95658bf4-76mh2\" (UID: \"f89ecfac-1ba1-4152-93c3-b97f9d27175e\") " pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.629211 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.629131 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" Jul 15 19:19:41.773290 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.773268 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c95658bf4-76mh2"] Jul 15 19:19:41.775395 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:19:41.775370 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89ecfac_1ba1_4152_93c3_b97f9d27175e.slice/crio-6ac6dc41de709b27755032132b36fdb595408b3cc264a49f698c8244ec1f2ad4 WatchSource:0}: Error finding container 6ac6dc41de709b27755032132b36fdb595408b3cc264a49f698c8244ec1f2ad4: Status 404 returned error can't find the container with id 6ac6dc41de709b27755032132b36fdb595408b3cc264a49f698c8244ec1f2ad4 Jul 15 19:19:41.777154 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:41.777139 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:19:42.340614 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:42.340580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:42.416377 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:42.416328 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:42.654012 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:42.653929 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" event={"ID":"f89ecfac-1ba1-4152-93c3-b97f9d27175e","Type":"ContainerStarted","Data":"6ac6dc41de709b27755032132b36fdb595408b3cc264a49f698c8244ec1f2ad4"} Jul 15 19:19:43.892828 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:43.892783 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:45.668661 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:45.668626 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" event={"ID":"f89ecfac-1ba1-4152-93c3-b97f9d27175e","Type":"ContainerStarted","Data":"5c2c7e019187ce285f9a9f44e0da35706fd460a783ffb678133d4941a4f9954b"} Jul 15 19:19:45.687604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:45.687558 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c95658bf4-76mh2" podStartSLOduration=1.44522086 podStartE2EDuration="4.687545104s" totalImagesPullingTime="3.242324244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:19:45.686824236 +0000 UTC m=+319.877702056" watchObservedRunningTime="2026-07-15 19:19:45.687545104 +0000 UTC m=+319.878422909" Jul 15 19:19:47.035865 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.035833 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9"] Jul 15 19:19:47.039320 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.039300 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.042715 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.042690 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 19:19:47.045497 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.045255 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gzqdv\"" Jul 15 19:19:47.046808 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.046791 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 19:19:47.049247 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.049233 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 19:19:47.067761 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.067740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9"] Jul 15 19:19:47.095320 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.095299 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56ce0f04-775a-4021-86be-fb81eb986c8d-cert\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.095430 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.095325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56ce0f04-775a-4021-86be-fb81eb986c8d-manager-config\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.095430 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.095419 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzqrt\" (UniqueName: \"kubernetes.io/projected/56ce0f04-775a-4021-86be-fb81eb986c8d-kube-api-access-tzqrt\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.095500 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.095464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56ce0f04-775a-4021-86be-fb81eb986c8d-metrics-cert\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.196099 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.196076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzqrt\" (UniqueName: \"kubernetes.io/projected/56ce0f04-775a-4021-86be-fb81eb986c8d-kube-api-access-tzqrt\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.196196 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.196110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56ce0f04-775a-4021-86be-fb81eb986c8d-metrics-cert\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.196196 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.196152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56ce0f04-775a-4021-86be-fb81eb986c8d-cert\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.196196 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.196169 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56ce0f04-775a-4021-86be-fb81eb986c8d-manager-config\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.196803 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.196773 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/56ce0f04-775a-4021-86be-fb81eb986c8d-manager-config\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.198617 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.198596 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/56ce0f04-775a-4021-86be-fb81eb986c8d-metrics-cert\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.198719 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.198691 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56ce0f04-775a-4021-86be-fb81eb986c8d-cert\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.208928 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.208906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzqrt\" (UniqueName: \"kubernetes.io/projected/56ce0f04-775a-4021-86be-fb81eb986c8d-kube-api-access-tzqrt\") pod \"lws-controller-manager-756b7fb4f9-9ddz9\" (UID: \"56ce0f04-775a-4021-86be-fb81eb986c8d\") " pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.347980 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.347912 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:47.523878 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.523850 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9"] Jul 15 19:19:47.526231 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:19:47.526204 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ce0f04_775a_4021_86be_fb81eb986c8d.slice/crio-c5042d9d61b2e3698d357012452102aab5ec90d6a24c302c8e58904c150c60a9 WatchSource:0}: Error finding container c5042d9d61b2e3698d357012452102aab5ec90d6a24c302c8e58904c150c60a9: Status 404 returned error can't find the container with id c5042d9d61b2e3698d357012452102aab5ec90d6a24c302c8e58904c150c60a9 Jul 15 19:19:47.677305 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.677242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" event={"ID":"56ce0f04-775a-4021-86be-fb81eb986c8d","Type":"ContainerStarted","Data":"c5042d9d61b2e3698d357012452102aab5ec90d6a24c302c8e58904c150c60a9"} Jul 15 19:19:47.731537 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.731512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:47.864547 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:47.864515 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:48.764152 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:48.764109 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:49.287132 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:49.287096 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:49.491011 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:49.490978 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:49.589175 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:49.589084 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:49.692930 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:49.692902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" event={"ID":"56ce0f04-775a-4021-86be-fb81eb986c8d","Type":"ContainerStarted","Data":"053eeb7578252d0d69066425942b6dfdc042dda2a2684844a80100ad9d0f311e"} Jul 15 19:19:49.693075 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:49.693006 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:19:49.727895 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:49.727849 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" podStartSLOduration=1.350219097 podStartE2EDuration="2.727834037s" totalImagesPullingTime="1.37761494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:19:49.726050507 +0000 UTC m=+323.916928312" watchObservedRunningTime="2026-07-15 19:19:49.727834037 +0000 UTC m=+323.918711844" Jul 15 19:19:51.304725 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:51.304695 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:51.394576 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:51.394544 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:51.630133 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:51.630038 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:51.722920 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:51.722885 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:52.665039 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:52.664998 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:52.756473 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:52.756442 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:52.858253 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:52.858218 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:56.692128 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:56.692088 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:19:56.792287 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:19:56.792254 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:00.699454 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:00.699428 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-756b7fb4f9-9ddz9" Jul 15 19:20:07.803525 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:07.803490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:07.904279 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:07.904251 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:09.764748 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:09.764719 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:09.842293 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:09.842265 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:39.374465 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:39.374372 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:39.468020 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:39.467987 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:39.630065 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:39.629975 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:20:55.142901 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.142867 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp"] Jul 15 19:20:55.146244 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.146226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.149068 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.149049 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:20:55.149174 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.149149 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fj8vl\"" Jul 15 19:20:55.149238 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.149198 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:20:55.157308 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.157286 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp"] Jul 15 19:20:55.199241 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.199212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg5wq\" (UniqueName: \"kubernetes.io/projected/47924db6-c526-4ee9-9124-6eb598ea17f5-kube-api-access-pg5wq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.199437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.199251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.199437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.199273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.300041 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.299997 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.300208 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.300097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg5wq\" (UniqueName: \"kubernetes.io/projected/47924db6-c526-4ee9-9124-6eb598ea17f5-kube-api-access-pg5wq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.300208 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.300128 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.300354 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.300326 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.300419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.300405 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.311237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.311209 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg5wq\" (UniqueName: \"kubernetes.io/projected/47924db6-c526-4ee9-9124-6eb598ea17f5-kube-api-access-pg5wq\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.455599 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.455513 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:55.593924 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.593902 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp"] Jul 15 19:20:55.596284 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:20:55.596255 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47924db6_c526_4ee9_9124_6eb598ea17f5.slice/crio-cafbcfab6dbf1ef5e9d426586a5a14cd33bb1acbf986a6542b64d3ea83d30a2b WatchSource:0}: Error finding container cafbcfab6dbf1ef5e9d426586a5a14cd33bb1acbf986a6542b64d3ea83d30a2b: Status 404 returned error can't find the container with id cafbcfab6dbf1ef5e9d426586a5a14cd33bb1acbf986a6542b64d3ea83d30a2b Jul 15 19:20:55.751987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.751959 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb"] Jul 15 19:20:55.755266 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.755250 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.770557 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.770538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb"] Jul 15 19:20:55.804148 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.804125 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhpkl\" (UniqueName: \"kubernetes.io/projected/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-kube-api-access-bhpkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.804263 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.804159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.804263 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.804182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.905053 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.905021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhpkl\" (UniqueName: \"kubernetes.io/projected/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-kube-api-access-bhpkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.905217 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.905060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.905217 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.905078 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.905465 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.905446 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.905506 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.905469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.916141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.916114 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhpkl\" (UniqueName: \"kubernetes.io/projected/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-kube-api-access-bhpkl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:55.975572 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.975541 2568 generic.go:356] "Generic (PLEG): container finished" podID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerID="da69b5bf992204247ae6ef948567157d2afc76a5c049940a9fa3f0215830048c" exitCode=0 Jul 15 19:20:55.975728 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.975628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" event={"ID":"47924db6-c526-4ee9-9124-6eb598ea17f5","Type":"ContainerDied","Data":"da69b5bf992204247ae6ef948567157d2afc76a5c049940a9fa3f0215830048c"} Jul 15 19:20:55.975728 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:55.975688 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" event={"ID":"47924db6-c526-4ee9-9124-6eb598ea17f5","Type":"ContainerStarted","Data":"cafbcfab6dbf1ef5e9d426586a5a14cd33bb1acbf986a6542b64d3ea83d30a2b"} Jul 15 19:20:56.064272 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.064198 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:20:56.192803 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.192776 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb"] Jul 15 19:20:56.193674 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:20:56.193641 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd4bbd29_ab05_4e10_817b_5acd62aeb8f3.slice/crio-81cd0f9848863268931d79afbaafadc7de8791d6836233fb9ac377985c414956 WatchSource:0}: Error finding container 81cd0f9848863268931d79afbaafadc7de8791d6836233fb9ac377985c414956: Status 404 returned error can't find the container with id 81cd0f9848863268931d79afbaafadc7de8791d6836233fb9ac377985c414956 Jul 15 19:20:56.364158 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.364131 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm"] Jul 15 19:20:56.367683 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.367663 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.383604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.383579 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm"] Jul 15 19:20:56.408496 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.408462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.408663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.408506 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.408663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.408555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgfsp\" (UniqueName: \"kubernetes.io/projected/d59775d5-69d8-4981-864c-c83b05a4d665-kube-api-access-qgfsp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.509114 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.509080 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.509284 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.509131 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgfsp\" (UniqueName: \"kubernetes.io/projected/d59775d5-69d8-4981-864c-c83b05a4d665-kube-api-access-qgfsp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.509284 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.509182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.509491 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.509470 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.509566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.509549 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.520058 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.520034 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgfsp\" (UniqueName: \"kubernetes.io/projected/d59775d5-69d8-4981-864c-c83b05a4d665-kube-api-access-qgfsp\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.676814 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.676747 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:20:56.758775 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.758748 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv"] Jul 15 19:20:56.762660 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.762179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.779070 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.778696 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv"] Jul 15 19:20:56.810555 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.810524 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk65v\" (UniqueName: \"kubernetes.io/projected/2db38e70-b464-4432-9d6c-565774c67c79-kube-api-access-qk65v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.810691 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.810576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.810691 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.810615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.822466 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.822445 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm"] Jul 15 19:20:56.842153 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:20:56.842126 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59775d5_69d8_4981_864c_c83b05a4d665.slice/crio-a15538201dd6f5df79e6054244d3d593b7f4cbf2eca1c9ce593a9e705a9ea28c WatchSource:0}: Error finding container a15538201dd6f5df79e6054244d3d593b7f4cbf2eca1c9ce593a9e705a9ea28c: Status 404 returned error can't find the container with id a15538201dd6f5df79e6054244d3d593b7f4cbf2eca1c9ce593a9e705a9ea28c Jul 15 19:20:56.911870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.911846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk65v\" (UniqueName: \"kubernetes.io/projected/2db38e70-b464-4432-9d6c-565774c67c79-kube-api-access-qk65v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.911971 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.911890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.911971 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.911920 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.912265 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.912248 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.912301 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.912280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.923506 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.923481 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk65v\" (UniqueName: \"kubernetes.io/projected/2db38e70-b464-4432-9d6c-565774c67c79-kube-api-access-qk65v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:56.981836 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.981809 2568 generic.go:356] "Generic (PLEG): container finished" podID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerID="74bbdafce678ab0d597fb8c3484bf5f605287d82b56485c3b505f4581e378e56" exitCode=0 Jul 15 19:20:56.982008 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.981878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" event={"ID":"47924db6-c526-4ee9-9124-6eb598ea17f5","Type":"ContainerDied","Data":"74bbdafce678ab0d597fb8c3484bf5f605287d82b56485c3b505f4581e378e56"} Jul 15 19:20:56.983543 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.983399 2568 generic.go:356] "Generic (PLEG): container finished" podID="d59775d5-69d8-4981-864c-c83b05a4d665" containerID="86e4f0146c6f3cee79e07e8d0fdbea39df449519e5f16322b67610cd87126e04" exitCode=0 Jul 15 19:20:56.983543 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.983464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" event={"ID":"d59775d5-69d8-4981-864c-c83b05a4d665","Type":"ContainerDied","Data":"86e4f0146c6f3cee79e07e8d0fdbea39df449519e5f16322b67610cd87126e04"} Jul 15 19:20:56.983543 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.983506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" event={"ID":"d59775d5-69d8-4981-864c-c83b05a4d665","Type":"ContainerStarted","Data":"a15538201dd6f5df79e6054244d3d593b7f4cbf2eca1c9ce593a9e705a9ea28c"} Jul 15 19:20:56.984942 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.984923 2568 generic.go:356] "Generic (PLEG): container finished" podID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerID="bb9cec45f8469c3edaf97f9271d14f579ad78068fbc75da92969b8a8dee56364" exitCode=0 Jul 15 19:20:56.985028 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.984954 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" event={"ID":"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3","Type":"ContainerDied","Data":"bb9cec45f8469c3edaf97f9271d14f579ad78068fbc75da92969b8a8dee56364"} Jul 15 19:20:56.985028 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:56.984979 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" event={"ID":"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3","Type":"ContainerStarted","Data":"81cd0f9848863268931d79afbaafadc7de8791d6836233fb9ac377985c414956"} Jul 15 19:20:57.074531 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.074503 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:20:57.224058 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.223961 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv"] Jul 15 19:20:57.229212 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:20:57.229181 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db38e70_b464_4432_9d6c_565774c67c79.slice/crio-cc235d317ca34d71ed87fb80e0c0b30b32ba62f736771f48e8acfa1640a3352c WatchSource:0}: Error finding container cc235d317ca34d71ed87fb80e0c0b30b32ba62f736771f48e8acfa1640a3352c: Status 404 returned error can't find the container with id cc235d317ca34d71ed87fb80e0c0b30b32ba62f736771f48e8acfa1640a3352c Jul 15 19:20:57.993048 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.993013 2568 generic.go:356] "Generic (PLEG): container finished" podID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerID="2e22e0461188f56346a2ceec6ac0c0f9a2c9b0d807c9f6970fd8dfe69aef9954" exitCode=0 Jul 15 19:20:57.993213 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.993090 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" event={"ID":"47924db6-c526-4ee9-9124-6eb598ea17f5","Type":"ContainerDied","Data":"2e22e0461188f56346a2ceec6ac0c0f9a2c9b0d807c9f6970fd8dfe69aef9954"} Jul 15 19:20:57.994845 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.994819 2568 generic.go:356] "Generic (PLEG): container finished" podID="2db38e70-b464-4432-9d6c-565774c67c79" containerID="c6f21135100a98e67068b303555e5060c648fa4173d01e3de50ad7a3276fefac" exitCode=0 Jul 15 19:20:57.994960 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.994852 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" event={"ID":"2db38e70-b464-4432-9d6c-565774c67c79","Type":"ContainerDied","Data":"c6f21135100a98e67068b303555e5060c648fa4173d01e3de50ad7a3276fefac"} Jul 15 19:20:57.994960 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.994900 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" event={"ID":"2db38e70-b464-4432-9d6c-565774c67c79","Type":"ContainerStarted","Data":"cc235d317ca34d71ed87fb80e0c0b30b32ba62f736771f48e8acfa1640a3352c"} Jul 15 19:20:57.997203 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.997180 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" event={"ID":"d59775d5-69d8-4981-864c-c83b05a4d665","Type":"ContainerStarted","Data":"8847fdf03f6ef8030e77a49b263d1257aab289ee92f8d874a65f8bc5c9e96f54"} Jul 15 19:20:57.999296 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:57.999268 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" event={"ID":"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3","Type":"ContainerStarted","Data":"8d83be5955d2a96d8f9dbabf72b76cc9d5fcc2d0300c38c96aa005e58ac40b6f"} Jul 15 19:20:59.005277 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.005244 2568 generic.go:356] "Generic (PLEG): container finished" podID="d59775d5-69d8-4981-864c-c83b05a4d665" containerID="8847fdf03f6ef8030e77a49b263d1257aab289ee92f8d874a65f8bc5c9e96f54" exitCode=0 Jul 15 19:20:59.005651 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.005328 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" event={"ID":"d59775d5-69d8-4981-864c-c83b05a4d665","Type":"ContainerDied","Data":"8847fdf03f6ef8030e77a49b263d1257aab289ee92f8d874a65f8bc5c9e96f54"} Jul 15 19:20:59.007132 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.007107 2568 generic.go:356] "Generic (PLEG): container finished" podID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerID="8d83be5955d2a96d8f9dbabf72b76cc9d5fcc2d0300c38c96aa005e58ac40b6f" exitCode=0 Jul 15 19:20:59.007218 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.007146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" event={"ID":"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3","Type":"ContainerDied","Data":"8d83be5955d2a96d8f9dbabf72b76cc9d5fcc2d0300c38c96aa005e58ac40b6f"} Jul 15 19:20:59.009040 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.009019 2568 generic.go:356] "Generic (PLEG): container finished" podID="2db38e70-b464-4432-9d6c-565774c67c79" containerID="901c8e32a6666e27e0cdb534d3d9da735e8dae794ff0fe1053ce4a18b4a78e01" exitCode=0 Jul 15 19:20:59.009137 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.009046 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" event={"ID":"2db38e70-b464-4432-9d6c-565774c67c79","Type":"ContainerDied","Data":"901c8e32a6666e27e0cdb534d3d9da735e8dae794ff0fe1053ce4a18b4a78e01"} Jul 15 19:20:59.149042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.149019 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:20:59.231550 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.231524 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg5wq\" (UniqueName: \"kubernetes.io/projected/47924db6-c526-4ee9-9124-6eb598ea17f5-kube-api-access-pg5wq\") pod \"47924db6-c526-4ee9-9124-6eb598ea17f5\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " Jul 15 19:20:59.231672 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.231613 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-util\") pod \"47924db6-c526-4ee9-9124-6eb598ea17f5\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " Jul 15 19:20:59.231672 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.231639 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-bundle\") pod \"47924db6-c526-4ee9-9124-6eb598ea17f5\" (UID: \"47924db6-c526-4ee9-9124-6eb598ea17f5\") " Jul 15 19:20:59.232311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.232279 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-bundle" (OuterVolumeSpecName: "bundle") pod "47924db6-c526-4ee9-9124-6eb598ea17f5" (UID: "47924db6-c526-4ee9-9124-6eb598ea17f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:20:59.233625 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.233603 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47924db6-c526-4ee9-9124-6eb598ea17f5-kube-api-access-pg5wq" (OuterVolumeSpecName: "kube-api-access-pg5wq") pod "47924db6-c526-4ee9-9124-6eb598ea17f5" (UID: "47924db6-c526-4ee9-9124-6eb598ea17f5"). InnerVolumeSpecName "kube-api-access-pg5wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:20:59.237001 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.236970 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-util" (OuterVolumeSpecName: "util") pod "47924db6-c526-4ee9-9124-6eb598ea17f5" (UID: "47924db6-c526-4ee9-9124-6eb598ea17f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:20:59.333108 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.333081 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-util\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:20:59.333108 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.333105 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47924db6-c526-4ee9-9124-6eb598ea17f5-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:20:59.333245 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:20:59.333120 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg5wq\" (UniqueName: \"kubernetes.io/projected/47924db6-c526-4ee9-9124-6eb598ea17f5-kube-api-access-pg5wq\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:00.016629 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.016599 2568 generic.go:356] "Generic (PLEG): container finished" podID="d59775d5-69d8-4981-864c-c83b05a4d665" containerID="4211d8cb6ffcb2ef499dc2a15c6df81fdfd81a337757dcea5fc591321435690c" exitCode=0 Jul 15 19:21:00.017034 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.016681 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" event={"ID":"d59775d5-69d8-4981-864c-c83b05a4d665","Type":"ContainerDied","Data":"4211d8cb6ffcb2ef499dc2a15c6df81fdfd81a337757dcea5fc591321435690c"} Jul 15 19:21:00.018551 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.018528 2568 generic.go:356] "Generic (PLEG): container finished" podID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerID="27297749a354f4143d1032fd97880a7819ec05ed63ed4eadc772654fbcaa6ef8" exitCode=0 Jul 15 19:21:00.018679 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.018607 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" event={"ID":"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3","Type":"ContainerDied","Data":"27297749a354f4143d1032fd97880a7819ec05ed63ed4eadc772654fbcaa6ef8"} Jul 15 19:21:00.020227 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.020209 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" event={"ID":"47924db6-c526-4ee9-9124-6eb598ea17f5","Type":"ContainerDied","Data":"cafbcfab6dbf1ef5e9d426586a5a14cd33bb1acbf986a6542b64d3ea83d30a2b"} Jul 15 19:21:00.020317 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.020241 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cafbcfab6dbf1ef5e9d426586a5a14cd33bb1acbf986a6542b64d3ea83d30a2b" Jul 15 19:21:00.020317 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.020215 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp" Jul 15 19:21:00.022470 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.022453 2568 generic.go:356] "Generic (PLEG): container finished" podID="2db38e70-b464-4432-9d6c-565774c67c79" containerID="181dcadebe9d478b01ce593405cafc7f2e646b6782c7834babd3a6f7016f01fc" exitCode=0 Jul 15 19:21:00.022566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:00.022488 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" event={"ID":"2db38e70-b464-4432-9d6c-565774c67c79","Type":"ContainerDied","Data":"181dcadebe9d478b01ce593405cafc7f2e646b6782c7834babd3a6f7016f01fc"} Jul 15 19:21:01.172143 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.172112 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:21:01.195939 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.195910 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:21:01.198980 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.198960 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:21:01.250080 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250058 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-bundle\") pod \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " Jul 15 19:21:01.250198 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250105 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhpkl\" (UniqueName: \"kubernetes.io/projected/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-kube-api-access-bhpkl\") pod \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " Jul 15 19:21:01.250198 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250138 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-util\") pod \"2db38e70-b464-4432-9d6c-565774c67c79\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " Jul 15 19:21:01.250198 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250174 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-bundle\") pod \"d59775d5-69d8-4981-864c-c83b05a4d665\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " Jul 15 19:21:01.250198 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250194 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qk65v\" (UniqueName: \"kubernetes.io/projected/2db38e70-b464-4432-9d6c-565774c67c79-kube-api-access-qk65v\") pod \"2db38e70-b464-4432-9d6c-565774c67c79\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " Jul 15 19:21:01.250433 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250223 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-bundle\") pod \"2db38e70-b464-4432-9d6c-565774c67c79\" (UID: \"2db38e70-b464-4432-9d6c-565774c67c79\") " Jul 15 19:21:01.250433 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250257 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-util\") pod \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\" (UID: \"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3\") " Jul 15 19:21:01.250433 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250290 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgfsp\" (UniqueName: \"kubernetes.io/projected/d59775d5-69d8-4981-864c-c83b05a4d665-kube-api-access-qgfsp\") pod \"d59775d5-69d8-4981-864c-c83b05a4d665\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " Jul 15 19:21:01.250433 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.250328 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-util\") pod \"d59775d5-69d8-4981-864c-c83b05a4d665\" (UID: \"d59775d5-69d8-4981-864c-c83b05a4d665\") " Jul 15 19:21:01.251121 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.251060 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-bundle" (OuterVolumeSpecName: "bundle") pod "2db38e70-b464-4432-9d6c-565774c67c79" (UID: "2db38e70-b464-4432-9d6c-565774c67c79"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:21:01.251121 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.251065 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-bundle" (OuterVolumeSpecName: "bundle") pod "cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" (UID: "cd4bbd29-ab05-4e10-817b-5acd62aeb8f3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:21:01.251362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.251290 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-bundle" (OuterVolumeSpecName: "bundle") pod "d59775d5-69d8-4981-864c-c83b05a4d665" (UID: "d59775d5-69d8-4981-864c-c83b05a4d665"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:21:01.253854 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.253811 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db38e70-b464-4432-9d6c-565774c67c79-kube-api-access-qk65v" (OuterVolumeSpecName: "kube-api-access-qk65v") pod "2db38e70-b464-4432-9d6c-565774c67c79" (UID: "2db38e70-b464-4432-9d6c-565774c67c79"). InnerVolumeSpecName "kube-api-access-qk65v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:21:01.254023 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.254006 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-kube-api-access-bhpkl" (OuterVolumeSpecName: "kube-api-access-bhpkl") pod "cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" (UID: "cd4bbd29-ab05-4e10-817b-5acd62aeb8f3"). InnerVolumeSpecName "kube-api-access-bhpkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:21:01.255861 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.255829 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59775d5-69d8-4981-864c-c83b05a4d665-kube-api-access-qgfsp" (OuterVolumeSpecName: "kube-api-access-qgfsp") pod "d59775d5-69d8-4981-864c-c83b05a4d665" (UID: "d59775d5-69d8-4981-864c-c83b05a4d665"). InnerVolumeSpecName "kube-api-access-qgfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:21:01.258548 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.258523 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-util" (OuterVolumeSpecName: "util") pod "d59775d5-69d8-4981-864c-c83b05a4d665" (UID: "d59775d5-69d8-4981-864c-c83b05a4d665"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:21:01.259133 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.259111 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-util" (OuterVolumeSpecName: "util") pod "cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" (UID: "cd4bbd29-ab05-4e10-817b-5acd62aeb8f3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:21:01.259729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.259699 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-util" (OuterVolumeSpecName: "util") pod "2db38e70-b464-4432-9d6c-565774c67c79" (UID: "2db38e70-b464-4432-9d6c-565774c67c79"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:21:01.351843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351776 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhpkl\" (UniqueName: \"kubernetes.io/projected/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-kube-api-access-bhpkl\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.351843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351799 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-util\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.351843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351809 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.351843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351817 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk65v\" (UniqueName: \"kubernetes.io/projected/2db38e70-b464-4432-9d6c-565774c67c79-kube-api-access-qk65v\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.351843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351827 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2db38e70-b464-4432-9d6c-565774c67c79-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.351843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351834 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-util\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.351843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351842 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgfsp\" (UniqueName: \"kubernetes.io/projected/d59775d5-69d8-4981-864c-c83b05a4d665-kube-api-access-qgfsp\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.352119 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351849 2568 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d59775d5-69d8-4981-864c-c83b05a4d665-util\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:01.352119 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:01.351857 2568 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cd4bbd29-ab05-4e10-817b-5acd62aeb8f3-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:02.032984 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.032956 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" Jul 15 19:21:02.033144 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.032957 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv" event={"ID":"2db38e70-b464-4432-9d6c-565774c67c79","Type":"ContainerDied","Data":"cc235d317ca34d71ed87fb80e0c0b30b32ba62f736771f48e8acfa1640a3352c"} Jul 15 19:21:02.033144 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.033081 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc235d317ca34d71ed87fb80e0c0b30b32ba62f736771f48e8acfa1640a3352c" Jul 15 19:21:02.034909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.034884 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" event={"ID":"d59775d5-69d8-4981-864c-c83b05a4d665","Type":"ContainerDied","Data":"a15538201dd6f5df79e6054244d3d593b7f4cbf2eca1c9ce593a9e705a9ea28c"} Jul 15 19:21:02.034909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.034900 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm" Jul 15 19:21:02.035103 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.034926 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a15538201dd6f5df79e6054244d3d593b7f4cbf2eca1c9ce593a9e705a9ea28c" Jul 15 19:21:02.036754 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.036733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" event={"ID":"cd4bbd29-ab05-4e10-817b-5acd62aeb8f3","Type":"ContainerDied","Data":"81cd0f9848863268931d79afbaafadc7de8791d6836233fb9ac377985c414956"} Jul 15 19:21:02.036884 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.036761 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb" Jul 15 19:21:02.036884 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:02.036763 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81cd0f9848863268931d79afbaafadc7de8791d6836233fb9ac377985c414956" Jul 15 19:21:05.201007 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.200976 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d74dbd896-pctjz"] Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201256 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201267 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201275 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201280 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201290 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db38e70-b464-4432-9d6c-565774c67c79" containerName="util" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201295 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db38e70-b464-4432-9d6c-565774c67c79" containerName="util" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201302 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db38e70-b464-4432-9d6c-565774c67c79" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201308 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db38e70-b464-4432-9d6c-565774c67c79" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201313 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201319 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201326 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db38e70-b464-4432-9d6c-565774c67c79" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201331 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db38e70-b464-4432-9d6c-565774c67c79" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201338 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59775d5-69d8-4981-864c-c83b05a4d665" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201357 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59775d5-69d8-4981-864c-c83b05a4d665" containerName="extract" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201363 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59775d5-69d8-4981-864c-c83b05a4d665" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201368 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59775d5-69d8-4981-864c-c83b05a4d665" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201376 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerName="util" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201381 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerName="util" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201388 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerName="util" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201393 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerName="util" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201400 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201405 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerName="pull" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201410 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59775d5-69d8-4981-864c-c83b05a4d665" containerName="util" Jul 15 19:21:05.201419 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201414 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59775d5-69d8-4981-864c-c83b05a4d665" containerName="util" Jul 15 19:21:05.202066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201464 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd4bbd29-ab05-4e10-817b-5acd62aeb8f3" containerName="extract" Jul 15 19:21:05.202066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201473 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59775d5-69d8-4981-864c-c83b05a4d665" containerName="extract" Jul 15 19:21:05.202066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201478 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db38e70-b464-4432-9d6c-565774c67c79" containerName="extract" Jul 15 19:21:05.202066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.201483 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="47924db6-c526-4ee9-9124-6eb598ea17f5" containerName="extract" Jul 15 19:21:05.205486 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.205465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.220999 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.220971 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d74dbd896-pctjz"] Jul 15 19:21:05.259913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.259881 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:05.282188 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.282161 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b109634-10fd-4ae5-8254-3750be26ac57-console-serving-cert\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.282288 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.282239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-oauth-serving-cert\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.282338 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.282320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-service-ca\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.282414 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.282362 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b109634-10fd-4ae5-8254-3750be26ac57-console-oauth-config\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.282495 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.282473 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqb6z\" (UniqueName: \"kubernetes.io/projected/4b109634-10fd-4ae5-8254-3750be26ac57-kube-api-access-kqb6z\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.282561 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.282546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-console-config\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.282595 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.282572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-trusted-ca-bundle\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.325793 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.325768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:05.383523 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.383499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-console-config\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.383606 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.383527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-trusted-ca-bundle\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.383606 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.383556 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b109634-10fd-4ae5-8254-3750be26ac57-console-serving-cert\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.383606 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.383586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-oauth-serving-cert\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.383727 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.383630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-service-ca\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.383727 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.383646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b109634-10fd-4ae5-8254-3750be26ac57-console-oauth-config\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.383727 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.383661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqb6z\" (UniqueName: \"kubernetes.io/projected/4b109634-10fd-4ae5-8254-3750be26ac57-kube-api-access-kqb6z\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.384362 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.384320 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-console-config\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.384461 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.384381 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-oauth-serving-cert\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.384461 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.384392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-service-ca\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.384461 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.384441 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b109634-10fd-4ae5-8254-3750be26ac57-trusted-ca-bundle\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.385923 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.385906 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b109634-10fd-4ae5-8254-3750be26ac57-console-serving-cert\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.386066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.386048 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4b109634-10fd-4ae5-8254-3750be26ac57-console-oauth-config\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.394055 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.394033 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqb6z\" (UniqueName: \"kubernetes.io/projected/4b109634-10fd-4ae5-8254-3750be26ac57-kube-api-access-kqb6z\") pod \"console-6d74dbd896-pctjz\" (UID: \"4b109634-10fd-4ae5-8254-3750be26ac57\") " pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.514934 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.514911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:05.645303 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:05.645277 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d74dbd896-pctjz"] Jul 15 19:21:05.646982 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:21:05.646951 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b109634_10fd_4ae5_8254_3750be26ac57.slice/crio-cd3e695c63c34ce6e7e131efd6f09fc891b1eef5144438ed3d5338e098b4b768 WatchSource:0}: Error finding container cd3e695c63c34ce6e7e131efd6f09fc891b1eef5144438ed3d5338e098b4b768: Status 404 returned error can't find the container with id cd3e695c63c34ce6e7e131efd6f09fc891b1eef5144438ed3d5338e098b4b768 Jul 15 19:21:06.055332 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:06.055298 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d74dbd896-pctjz" event={"ID":"4b109634-10fd-4ae5-8254-3750be26ac57","Type":"ContainerStarted","Data":"e9d0804a0b3fcbf551eba3c7f0dcd19a64ca014e7db6e4fdd37873e4619b1455"} Jul 15 19:21:06.055518 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:06.055360 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d74dbd896-pctjz" event={"ID":"4b109634-10fd-4ae5-8254-3750be26ac57","Type":"ContainerStarted","Data":"cd3e695c63c34ce6e7e131efd6f09fc891b1eef5144438ed3d5338e098b4b768"} Jul 15 19:21:06.078120 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:06.078067 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d74dbd896-pctjz" podStartSLOduration=1.078052526 podStartE2EDuration="1.078052526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:06.075671396 +0000 UTC m=+400.266549199" watchObservedRunningTime="2026-07-15 19:21:06.078052526 +0000 UTC m=+400.268930331" Jul 15 19:21:08.712744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:08.712709 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:12.921509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:12.921475 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:15.515393 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:15.515357 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:15.515828 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:15.515417 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:15.520045 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:15.520022 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:15.527168 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:15.527147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:16.103020 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:16.102992 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d74dbd896-pctjz" Jul 15 19:21:16.184145 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:16.184116 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b57fcc84-r4mvs"] Jul 15 19:21:16.334204 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:16.334164 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:16.429097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:16.429022 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:16.538802 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:16.538774 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:17.178080 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:17.178050 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:18.549066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:18.549033 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:20.138067 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:20.138040 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:20.376710 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:20.376677 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:20.740493 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:20.740463 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:22.966042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:22.966012 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:23.060558 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:23.060534 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:25.131729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:25.131692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:25.236517 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:25.236487 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:27.128631 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:27.128602 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:27.225190 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:27.225158 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:30.242016 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.241982 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5"] Jul 15 19:21:30.245738 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.245719 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.249113 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.249093 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 19:21:30.249601 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.249583 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 19:21:30.249706 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.249590 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fj8vl\"" Jul 15 19:21:30.250821 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.250800 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:30.252521 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.252487 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 19:21:30.252639 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.252617 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 19:21:30.258916 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.258896 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5"] Jul 15 19:21:30.380109 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.380076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b89624c-25c9-4517-a5ed-0e455ca50ad3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.380109 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.380111 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b89624c-25c9-4517-a5ed-0e455ca50ad3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.380386 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.380194 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpx2\" (UniqueName: \"kubernetes.io/projected/5b89624c-25c9-4517-a5ed-0e455ca50ad3-kube-api-access-zmpx2\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.481476 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.481435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpx2\" (UniqueName: \"kubernetes.io/projected/5b89624c-25c9-4517-a5ed-0e455ca50ad3-kube-api-access-zmpx2\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.481664 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.481541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b89624c-25c9-4517-a5ed-0e455ca50ad3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.481664 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.481578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b89624c-25c9-4517-a5ed-0e455ca50ad3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.481797 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:21:30.481703 2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 15 19:21:30.481797 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:21:30.481793 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b89624c-25c9-4517-a5ed-0e455ca50ad3-plugin-serving-cert podName:5b89624c-25c9-4517-a5ed-0e455ca50ad3 nodeName:}" failed. No retries permitted until 2026-07-15 19:21:30.981769826 +0000 UTC m=+425.172647625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5b89624c-25c9-4517-a5ed-0e455ca50ad3-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-rvzp5" (UID: "5b89624c-25c9-4517-a5ed-0e455ca50ad3") : secret "plugin-serving-cert" not found Jul 15 19:21:30.482327 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.482306 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b89624c-25c9-4517-a5ed-0e455ca50ad3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.492884 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.492822 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpx2\" (UniqueName: \"kubernetes.io/projected/5b89624c-25c9-4517-a5ed-0e455ca50ad3-kube-api-access-zmpx2\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.749422 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.749325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:30.986870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.986832 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b89624c-25c9-4517-a5ed-0e455ca50ad3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:30.989170 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:30.989141 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b89624c-25c9-4517-a5ed-0e455ca50ad3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-rvzp5\" (UID: \"5b89624c-25c9-4517-a5ed-0e455ca50ad3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:31.157122 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:31.157034 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" Jul 15 19:21:31.298645 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:31.298577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5"] Jul 15 19:21:31.301301 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:21:31.301268 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b89624c_25c9_4517_a5ed_0e455ca50ad3.slice/crio-1a5f340b8fc0b0e8e8d7ec0ce6cc5ae6bb733e87ca442c4f03b7bc3faf87833e WatchSource:0}: Error finding container 1a5f340b8fc0b0e8e8d7ec0ce6cc5ae6bb733e87ca442c4f03b7bc3faf87833e: Status 404 returned error can't find the container with id 1a5f340b8fc0b0e8e8d7ec0ce6cc5ae6bb733e87ca442c4f03b7bc3faf87833e Jul 15 19:21:32.173498 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:32.173458 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" event={"ID":"5b89624c-25c9-4517-a5ed-0e455ca50ad3","Type":"ContainerStarted","Data":"1a5f340b8fc0b0e8e8d7ec0ce6cc5ae6bb733e87ca442c4f03b7bc3faf87833e"} Jul 15 19:21:32.330855 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:32.330823 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:32.431659 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:32.431578 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:40.602032 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:40.601992 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:40.705109 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:40.705054 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:41.204571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:41.204523 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b57fcc84-r4mvs" podUID="8f6df6b4-a11e-4582-840c-132fed12cbf1" containerName="console" containerID="cri-o://7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e" gracePeriod=15 Jul 15 19:21:43.166380 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:43.166318 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:43.269386 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:43.269331 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:46.373480 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:46.373442 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:46.451842 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:46.451794 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:46.667391 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:46.667197 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:46.766774 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:46.766740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:50.838504 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:50.838418 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:50.935247 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:50.935218 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:21:51.158437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:51.158323 2568 patch_prober.go:28] interesting pod/console-6b57fcc84-r4mvs container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" start-of-body= Jul 15 19:21:51.158437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:51.158407 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b57fcc84-r4mvs" podUID="8f6df6b4-a11e-4582-840c-132fed12cbf1" containerName="console" probeResult="failure" output="Get \"https://10.133.0.18:8443/health\": dial tcp 10.133.0.18:8443: connect: connection refused" Jul 15 19:21:54.188022 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.187999 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:21:54.284630 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284572 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-oauth-config\") pod \"8f6df6b4-a11e-4582-840c-132fed12cbf1\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " Jul 15 19:21:54.284630 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284601 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-serving-cert\") pod \"8f6df6b4-a11e-4582-840c-132fed12cbf1\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " Jul 15 19:21:54.284630 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284618 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-config\") pod \"8f6df6b4-a11e-4582-840c-132fed12cbf1\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " Jul 15 19:21:54.284839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284659 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csmsc\" (UniqueName: \"kubernetes.io/projected/8f6df6b4-a11e-4582-840c-132fed12cbf1-kube-api-access-csmsc\") pod \"8f6df6b4-a11e-4582-840c-132fed12cbf1\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " Jul 15 19:21:54.284839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284683 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-service-ca\") pod \"8f6df6b4-a11e-4582-840c-132fed12cbf1\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " Jul 15 19:21:54.284839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284716 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-trusted-ca-bundle\") pod \"8f6df6b4-a11e-4582-840c-132fed12cbf1\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " Jul 15 19:21:54.284839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284743 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-oauth-serving-cert\") pod \"8f6df6b4-a11e-4582-840c-132fed12cbf1\" (UID: \"8f6df6b4-a11e-4582-840c-132fed12cbf1\") " Jul 15 19:21:54.285032 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.284996 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-config" (OuterVolumeSpecName: "console-config") pod "8f6df6b4-a11e-4582-840c-132fed12cbf1" (UID: "8f6df6b4-a11e-4582-840c-132fed12cbf1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:21:54.285150 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.285132 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f6df6b4-a11e-4582-840c-132fed12cbf1" (UID: "8f6df6b4-a11e-4582-840c-132fed12cbf1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:21:54.285271 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.285249 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f6df6b4-a11e-4582-840c-132fed12cbf1" (UID: "8f6df6b4-a11e-4582-840c-132fed12cbf1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:21:54.285536 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.285508 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f6df6b4-a11e-4582-840c-132fed12cbf1" (UID: "8f6df6b4-a11e-4582-840c-132fed12cbf1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:21:54.286775 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.286751 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6df6b4-a11e-4582-840c-132fed12cbf1-kube-api-access-csmsc" (OuterVolumeSpecName: "kube-api-access-csmsc") pod "8f6df6b4-a11e-4582-840c-132fed12cbf1" (UID: "8f6df6b4-a11e-4582-840c-132fed12cbf1"). InnerVolumeSpecName "kube-api-access-csmsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:21:54.286857 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.286814 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f6df6b4-a11e-4582-840c-132fed12cbf1" (UID: "8f6df6b4-a11e-4582-840c-132fed12cbf1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:21:54.287008 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.286984 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f6df6b4-a11e-4582-840c-132fed12cbf1" (UID: "8f6df6b4-a11e-4582-840c-132fed12cbf1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:21:54.295940 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.295907 2568 generic.go:356] "Generic (PLEG): container finished" podID="8f6df6b4-a11e-4582-840c-132fed12cbf1" containerID="7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e" exitCode=0 Jul 15 19:21:54.296024 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.295954 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b57fcc84-r4mvs" Jul 15 19:21:54.296024 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.295988 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57fcc84-r4mvs" event={"ID":"8f6df6b4-a11e-4582-840c-132fed12cbf1","Type":"ContainerDied","Data":"7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e"} Jul 15 19:21:54.296090 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.296029 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b57fcc84-r4mvs" event={"ID":"8f6df6b4-a11e-4582-840c-132fed12cbf1","Type":"ContainerDied","Data":"19124f926caa0077ea7ffe6cde100399dc6257d4448b74dd30d873041ae2339c"} Jul 15 19:21:54.296090 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.296055 2568 scope.go:117] "RemoveContainer" containerID="7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e" Jul 15 19:21:54.324526 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.324497 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b57fcc84-r4mvs"] Jul 15 19:21:54.337227 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.337203 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b57fcc84-r4mvs"] Jul 15 19:21:54.385162 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.385140 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-service-ca\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:54.385162 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.385159 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-trusted-ca-bundle\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:54.385294 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.385169 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-oauth-serving-cert\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:54.385294 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.385178 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-oauth-config\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:54.385294 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.385186 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-serving-cert\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:54.385294 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.385194 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f6df6b4-a11e-4582-840c-132fed12cbf1-console-config\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:54.385294 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.385203 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csmsc\" (UniqueName: \"kubernetes.io/projected/8f6df6b4-a11e-4582-840c-132fed12cbf1-kube-api-access-csmsc\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:21:54.448401 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.448375 2568 scope.go:117] "RemoveContainer" containerID="7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e" Jul 15 19:21:54.448761 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:21:54.448731 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e\": container with ID starting with 7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e not found: ID does not exist" containerID="7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e" Jul 15 19:21:54.448847 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:54.448763 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e"} err="failed to get container status \"7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e\": rpc error: code = NotFound desc = could not find container \"7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e\": container with ID starting with 7e38d08877c6bebe137002cc3eec9cdc10c365b47ba064dbd84ca9f7103a872e not found: ID does not exist" Jul 15 19:21:55.302802 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:55.302761 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" event={"ID":"5b89624c-25c9-4517-a5ed-0e455ca50ad3","Type":"ContainerStarted","Data":"09dd680035f06acce48c865ab2dc530052cec486647b4dc0da790632e9bd07a8"} Jul 15 19:21:55.326437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:55.326379 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-rvzp5" podStartSLOduration=2.148037385 podStartE2EDuration="25.326364808s" totalImagesPullingTime="23.178327423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:21:55.324123107 +0000 UTC m=+449.515000913" watchObservedRunningTime="2026-07-15 19:21:55.326364808 +0000 UTC m=+449.517242612" Jul 15 19:21:56.229016 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:21:56.228982 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f6df6b4-a11e-4582-840c-132fed12cbf1" path="/var/lib/kubelet/pods/8f6df6b4-a11e-4582-840c-132fed12cbf1/volumes" Jul 15 19:22:00.961517 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:00.961452 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:01.060936 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:01.060905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:10.115000 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:10.114963 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:10.677851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:10.677819 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:12.859029 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:12.858996 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:12.968901 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:12.968868 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:13.038556 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.038526 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:13.295642 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.295616 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:22:13.295965 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.295949 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6df6b4-a11e-4582-840c-132fed12cbf1" containerName="console" Jul 15 19:22:13.296045 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.295969 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6df6b4-a11e-4582-840c-132fed12cbf1" containerName="console" Jul 15 19:22:13.296097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.296062 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f6df6b4-a11e-4582-840c-132fed12cbf1" containerName="console" Jul 15 19:22:13.299117 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.299100 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.302697 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.302682 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 19:22:13.318621 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.318591 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:22:13.321214 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.321189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9c8c178-d752-4caa-b653-c549fb595ef4-config-file\") pod \"limitador-limitador-54d6d988d5-l2vjt\" (UID: \"e9c8c178-d752-4caa-b653-c549fb595ef4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.321320 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.321302 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvw4\" (UniqueName: \"kubernetes.io/projected/e9c8c178-d752-4caa-b653-c549fb595ef4-kube-api-access-9tvw4\") pod \"limitador-limitador-54d6d988d5-l2vjt\" (UID: \"e9c8c178-d752-4caa-b653-c549fb595ef4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.358131 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.358105 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:22:13.422696 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.422667 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvw4\" (UniqueName: \"kubernetes.io/projected/e9c8c178-d752-4caa-b653-c549fb595ef4-kube-api-access-9tvw4\") pod \"limitador-limitador-54d6d988d5-l2vjt\" (UID: \"e9c8c178-d752-4caa-b653-c549fb595ef4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.422849 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.422724 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9c8c178-d752-4caa-b653-c549fb595ef4-config-file\") pod \"limitador-limitador-54d6d988d5-l2vjt\" (UID: \"e9c8c178-d752-4caa-b653-c549fb595ef4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.423269 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.423232 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e9c8c178-d752-4caa-b653-c549fb595ef4-config-file\") pod \"limitador-limitador-54d6d988d5-l2vjt\" (UID: \"e9c8c178-d752-4caa-b653-c549fb595ef4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.437073 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.437049 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvw4\" (UniqueName: \"kubernetes.io/projected/e9c8c178-d752-4caa-b653-c549fb595ef4-kube-api-access-9tvw4\") pod \"limitador-limitador-54d6d988d5-l2vjt\" (UID: \"e9c8c178-d752-4caa-b653-c549fb595ef4\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.608770 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.608705 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:13.741441 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.741370 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:22:13.746519 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:22:13.746496 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c8c178_d752_4caa_b653_c549fb595ef4.slice/crio-bcdeda65c1a2f662dff91e80063c4b8fc72282a0b36656e956e3567cbb072d63 WatchSource:0}: Error finding container bcdeda65c1a2f662dff91e80063c4b8fc72282a0b36656e956e3567cbb072d63: Status 404 returned error can't find the container with id bcdeda65c1a2f662dff91e80063c4b8fc72282a0b36656e956e3567cbb072d63 Jul 15 19:22:13.873533 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.873471 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:13.980311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:13.980279 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:14.399231 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:14.399149 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" event={"ID":"e9c8c178-d752-4caa-b653-c549fb595ef4","Type":"ContainerStarted","Data":"bcdeda65c1a2f662dff91e80063c4b8fc72282a0b36656e956e3567cbb072d63"} Jul 15 19:22:16.411175 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:16.411142 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" event={"ID":"e9c8c178-d752-4caa-b653-c549fb595ef4","Type":"ContainerStarted","Data":"8f943657dc074b3d7c93ef5ebdfe7d75ae5470703a6e70dfdcdc479d58ac4f7b"} Jul 15 19:22:16.411481 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:16.411269 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:16.448739 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:16.448698 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" podStartSLOduration=0.904057406 podStartE2EDuration="3.448685256s" totalImagesPullingTime="2.54462785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:22:16.448403177 +0000 UTC m=+470.639281001" watchObservedRunningTime="2026-07-15 19:22:16.448685256 +0000 UTC m=+470.639563059" Jul 15 19:22:27.416847 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:27.416820 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l2vjt" Jul 15 19:22:28.954690 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:28.954646 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:29.046562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:29.046533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:29.483426 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:29.483396 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:41.494801 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:41.494764 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:41.726578 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:41.726544 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:42.044846 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.044818 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f5d69c875-j9bdz"] Jul 15 19:22:42.127932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.127894 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f5d69c875-j9bdz"] Jul 15 19:22:42.128081 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.127994 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d69c875-j9bdz" Jul 15 19:22:42.130674 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.130655 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fdgt2\"" Jul 15 19:22:42.177071 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.177048 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5d69c875-j9bdz"] Jul 15 19:22:42.177247 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:22:42.177230 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-n9s7p], unattached volumes=[], failed to process volumes=[kube-api-access-n9s7p]: context canceled" pod="kuadrant-system/authorino-5f5d69c875-j9bdz" podUID="e5fed2d0-bfd4-4435-b4d8-be6ee54322b4" Jul 15 19:22:42.247376 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.247337 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9s7p\" (UniqueName: \"kubernetes.io/projected/e5fed2d0-bfd4-4435-b4d8-be6ee54322b4-kube-api-access-n9s7p\") pod \"authorino-5f5d69c875-j9bdz\" (UID: \"e5fed2d0-bfd4-4435-b4d8-be6ee54322b4\") " pod="kuadrant-system/authorino-5f5d69c875-j9bdz" Jul 15 19:22:42.348511 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.348450 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9s7p\" (UniqueName: \"kubernetes.io/projected/e5fed2d0-bfd4-4435-b4d8-be6ee54322b4-kube-api-access-n9s7p\") pod \"authorino-5f5d69c875-j9bdz\" (UID: \"e5fed2d0-bfd4-4435-b4d8-be6ee54322b4\") " pod="kuadrant-system/authorino-5f5d69c875-j9bdz" Jul 15 19:22:42.362171 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.362148 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9s7p\" (UniqueName: \"kubernetes.io/projected/e5fed2d0-bfd4-4435-b4d8-be6ee54322b4-kube-api-access-n9s7p\") pod \"authorino-5f5d69c875-j9bdz\" (UID: \"e5fed2d0-bfd4-4435-b4d8-be6ee54322b4\") " pod="kuadrant-system/authorino-5f5d69c875-j9bdz" Jul 15 19:22:42.532747 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.532723 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d69c875-j9bdz" Jul 15 19:22:42.539721 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.539702 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d69c875-j9bdz" Jul 15 19:22:42.650636 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.650580 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9s7p\" (UniqueName: \"kubernetes.io/projected/e5fed2d0-bfd4-4435-b4d8-be6ee54322b4-kube-api-access-n9s7p\") pod \"e5fed2d0-bfd4-4435-b4d8-be6ee54322b4\" (UID: \"e5fed2d0-bfd4-4435-b4d8-be6ee54322b4\") " Jul 15 19:22:42.652663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.652639 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5fed2d0-bfd4-4435-b4d8-be6ee54322b4-kube-api-access-n9s7p" (OuterVolumeSpecName: "kube-api-access-n9s7p") pod "e5fed2d0-bfd4-4435-b4d8-be6ee54322b4" (UID: "e5fed2d0-bfd4-4435-b4d8-be6ee54322b4"). InnerVolumeSpecName "kube-api-access-n9s7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:22:42.751246 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:42.751223 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9s7p\" (UniqueName: \"kubernetes.io/projected/e5fed2d0-bfd4-4435-b4d8-be6ee54322b4-kube-api-access-n9s7p\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:22:43.538244 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:43.538215 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f5d69c875-j9bdz" Jul 15 19:22:43.574616 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:43.574591 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f5d69c875-j9bdz"] Jul 15 19:22:43.580297 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:43.580278 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f5d69c875-j9bdz"] Jul 15 19:22:44.228974 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:44.228929 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5fed2d0-bfd4-4435-b4d8-be6ee54322b4" path="/var/lib/kubelet/pods/e5fed2d0-bfd4-4435-b4d8-be6ee54322b4/volumes" Jul 15 19:22:48.876438 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:48.876401 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:48.966575 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:48.966545 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:49.029039 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:49.029015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:49.450942 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:49.450908 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:53.018708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:53.018677 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:22:53.110663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:22:53.110636 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:10.132679 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:10.132644 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:18.303247 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:18.303177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:18.405107 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:18.405075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:21.790753 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:21.790719 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:25.899210 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:25.899176 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:25.986890 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:25.986856 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:26.443892 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:26.443858 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:26.560848 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:26.560817 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:27.213089 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:27.213051 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:27.311179 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:27.311140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:27.480100 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:27.480017 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:23:36.409582 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:36.409552 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:36.509388 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:36.509338 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:37.247415 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:37.247384 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:37.313659 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:37.313612 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:54.334086 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:54.334048 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:23:54.881128 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:54.881093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:23:59.254963 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:23:59.254929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:24:00.228206 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:00.228178 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:24:09.235169 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:09.235137 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:24:10.507189 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:10.507152 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:24:25.652333 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:25.652304 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:24:27.401186 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:27.401156 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:24:36.817117 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:36.817082 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:24:37.192158 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.192082 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj"] Jul 15 19:24:37.195255 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.195239 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.198753 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.198733 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 19:24:37.200111 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.200094 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 19:24:37.200201 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.200153 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 19:24:37.200259 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.200203 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vclgr\"" Jul 15 19:24:37.205587 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.205566 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj"] Jul 15 19:24:37.279469 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.279433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89kbx\" (UniqueName: \"kubernetes.io/projected/50c03f19-0056-48ca-81e8-5f05fe0f1b79-kube-api-access-89kbx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.279627 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.279490 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c03f19-0056-48ca-81e8-5f05fe0f1b79-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.279627 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.279551 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.279627 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.279583 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.279627 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.279603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.279799 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.279631 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.380772 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.380738 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c03f19-0056-48ca-81e8-5f05fe0f1b79-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.380886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.380784 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.380886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.380814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.380886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.380840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.380886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.380879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.381316 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.380927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89kbx\" (UniqueName: \"kubernetes.io/projected/50c03f19-0056-48ca-81e8-5f05fe0f1b79-kube-api-access-89kbx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.381316 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.381208 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.381316 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.381225 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.381316 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.381296 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.383104 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.383085 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50c03f19-0056-48ca-81e8-5f05fe0f1b79-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.383277 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.383258 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50c03f19-0056-48ca-81e8-5f05fe0f1b79-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.395974 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.395952 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89kbx\" (UniqueName: \"kubernetes.io/projected/50c03f19-0056-48ca-81e8-5f05fe0f1b79-kube-api-access-89kbx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-gwwdj\" (UID: \"50c03f19-0056-48ca-81e8-5f05fe0f1b79\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.505962 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.505937 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:37.641600 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:37.641573 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj"] Jul 15 19:24:37.643550 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:24:37.643527 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c03f19_0056_48ca_81e8_5f05fe0f1b79.slice/crio-94ef1a64f7611dedf06b221e3f4393e5b4f838aa6f6ccf19f6a469f1c4165a98 WatchSource:0}: Error finding container 94ef1a64f7611dedf06b221e3f4393e5b4f838aa6f6ccf19f6a469f1c4165a98: Status 404 returned error can't find the container with id 94ef1a64f7611dedf06b221e3f4393e5b4f838aa6f6ccf19f6a469f1c4165a98 Jul 15 19:24:38.070746 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:38.070710 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" event={"ID":"50c03f19-0056-48ca-81e8-5f05fe0f1b79","Type":"ContainerStarted","Data":"94ef1a64f7611dedf06b221e3f4393e5b4f838aa6f6ccf19f6a469f1c4165a98"} Jul 15 19:24:38.745955 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:38.745927 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:24:39.077023 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:39.076991 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" event={"ID":"50c03f19-0056-48ca-81e8-5f05fe0f1b79","Type":"ContainerStarted","Data":"525ad5167551aa309735eedf4218e46a7b51bd9c9bee05d8b22eeaab0211dbad"} Jul 15 19:24:39.099136 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:39.099090 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" podStartSLOduration=1.035842919 podStartE2EDuration="2.099075286s" totalImagesPullingTime="1.063232367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:24:39.098133382 +0000 UTC m=+613.289011207" watchObservedRunningTime="2026-07-15 19:24:39.099075286 +0000 UTC m=+613.289953090" Jul 15 19:24:39.410864 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:39.410783 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:24:47.507119 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:47.507088 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:47.507601 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:47.507138 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:47.519275 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:47.519252 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:48.128871 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:48.128849 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-gwwdj" Jul 15 19:24:48.136433 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:48.136395 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:24:50.823465 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:24:50.823432 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:25:58.205101 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:25:58.205045 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:58.654227 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:25:58.654193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:59.405027 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:25:59.404998 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:25:59.501615 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:25:59.501584 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:02.375152 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:02.375120 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:26.810431 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:26.810309 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:26:28.232917 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:28.232886 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:28.339638 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:28.339605 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:31.556796 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:31.556758 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:26:38.699423 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:38.699392 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:26:48.615280 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:48.615250 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:26:57.795513 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:26:57.795479 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:27:09.198711 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:09.198674 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:27:10.565701 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:10.565667 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:12.222535 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:12.222496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:39.096643 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:39.096605 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:27:53.392926 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:53.392837 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:27:59.103657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:59.103616 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:59.205046 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:59.205008 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:27:59.625091 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:27:59.625057 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:06.340883 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:06.340848 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:06.399252 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:06.399227 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:06.492874 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:06.492842 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:06.598680 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:06.598609 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:06.782722 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:06.782694 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:12.971134 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:12.971097 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:13.032221 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:13.032195 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:28:31.999956 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:31.999926 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:28:41.403835 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:41.403798 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:28:49.298765 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:49.298731 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:28:57.680700 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:28:57.680667 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:29:07.801458 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:29:07.801424 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:29:17.602913 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:29:17.602874 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:29:21.593161 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:29:21.593121 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:29:30.607998 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:29:30.607965 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:29:40.595447 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:29:40.595418 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:30:05.014670 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:30:05.014638 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:30:16.712373 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:30:16.712326 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:30:37.402832 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:30:37.402793 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:30:42.661471 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:30:42.661434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:49.124588 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:30:49.124501 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:30:50.504865 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:30:50.504830 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:31:06.693369 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:31:06.693310 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:31:23.817688 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:31:23.817650 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:31:34.290685 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:31:34.290649 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:31:48.595898 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:31:48.595860 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:32:25.861510 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:32:25.861472 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:32:25.945717 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:32:25.945683 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:33:34.112508 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:33:34.112473 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:33:43.994330 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:33:43.994302 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:33:57.597835 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:33:57.597750 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:34:08.712974 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:34:08.712939 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:34:19.301028 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:34:19.300996 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:34:30.203990 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:34:30.203947 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:34:39.206229 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:34:39.206193 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:34:50.748413 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:34:50.748381 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:35:01.403711 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:35:01.403674 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:35:09.995622 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:35:09.995585 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:35:21.242883 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:35:21.242785 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:35:42.718272 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:35:42.718240 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:35:52.608984 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:35:52.608948 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:36:02.008827 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:36:02.008793 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:36:12.542212 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:36:12.542179 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:36:30.712973 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:36:30.712942 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:36:40.601075 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:36:40.601042 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:36:50.103396 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:36:50.103304 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:37:00.614240 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:00.614206 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:37:09.816210 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:09.816179 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:37:17.044902 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:17.044867 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:37:19.297991 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:19.297952 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:37:33.403953 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:33.403911 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:37:43.197958 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:43.197924 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:37:54.174975 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:54.174923 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:37:58.204920 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:37:58.204886 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:38:07.405236 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:38:07.405198 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:38:08.159338 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:38:08.159298 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:08.257611 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:38:08.257578 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:38:19.914845 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:38:19.914762 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:38:37.000122 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:38:37.000088 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:38:48.897191 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:38:48.897155 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:38:58.438330 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:38:58.438294 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:39:00.510326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:00.510277 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:01.220606 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:01.220572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:08.931271 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:08.931237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:11.189287 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:11.189242 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:12.697592 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:12.697561 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:39:18.201782 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:18.201751 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:22.173687 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:22.173653 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:22.206089 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:22.206058 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:39:25.264258 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:25.264224 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:25.355809 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:25.355777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:25.714781 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:25.714683 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:32.400586 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:32.400546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:32.458406 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:32.458373 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:32.619520 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:32.619487 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:32.716006 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:32.715940 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:33.853516 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:33.853474 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:33.933006 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:33.932964 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:34.031661 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:34.031619 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:34.129574 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:34.129496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:34.557971 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:34.557945 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:34.655224 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:34.655187 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:35.093843 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:35.093806 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:39:36.588326 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:36.588292 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:39:46.989860 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:39:46.989828 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:40:49.447395 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:40:49.447338 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:40:52.400235 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:40:52.400203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:40:54.044751 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:40:54.044717 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:00.714396 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:00.714339 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:00.778358 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:00.778317 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:04.230899 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:04.230862 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:04.672042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:04.672003 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:05.139051 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:05.139010 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:05.200925 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:05.200894 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:06.101365 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:06.101312 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:06.210837 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:06.210797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:06.731629 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:06.731592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:06.851788 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:06.851755 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:06.965839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:06.965808 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:13.195135 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:13.195100 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:41:15.916067 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:15.916035 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:15.977141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:15.977108 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:41:19.401910 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:41:19.401875 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:10.763244 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:10.763206 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:10.880684 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:10.880644 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:11.351516 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:11.351486 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:13.713707 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:13.713674 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:42:17.473103 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:17.473073 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:17.601631 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:17.601596 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:17.670743 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:17.670710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:17.873806 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:17.873768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:17.974232 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:17.974194 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:19.000480 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:19.000446 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:20.172918 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:20.172878 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:20.958933 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:20.958888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:21.510443 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:21.510415 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:42:37.647131 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:37.647098 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:37.748505 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:37.748476 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:41.047510 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:41.047468 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:42:41.148476 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:42:41.148439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:24.744984 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:24.744954 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:27.601470 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:27.601432 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:43:29.259831 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:29.259788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:36.024850 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:36.024814 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:36.090741 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:36.090710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:40.366794 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:40.366756 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:40.696735 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:40.696690 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:41.813034 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:41.812999 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:41.908384 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:41.908321 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:41.979997 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:41.979967 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:42.573520 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:42.573488 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:42.674705 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:42.674677 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:42.776897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:42.776865 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:44.495857 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:44.495828 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:43:47.897800 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:47.897768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:47.998787 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:47.998757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:43:51.625856 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:43:51.625819 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:46.862193 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:44:46.862156 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:50.007145 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:44:50.007109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:44:51.404271 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:44:51.404220 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:57.543312 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:44:57.543276 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:44:57.644409 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:44:57.644373 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:01.628302 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:01.628262 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:02.208545 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:02.208514 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:02.456697 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:02.456665 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:02.563255 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:02.563225 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:02.898753 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:02.898673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:04.106139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:04.106106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:08.289541 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:08.289506 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.499133 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:10.499096 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.577215 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:10.577181 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.644905 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:10.644870 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.744592 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:10.744552 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:10.848744 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:10.848652 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:11.246611 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:11.246574 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:37.253759 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:37.253724 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:43.593066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:43.593010 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:43.657649 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:43.657620 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:43.736314 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:43.736290 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:43.831825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:43.831796 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:45:59.705961 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:45:59.705876 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:06.278797 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:06.278765 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:06.383885 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:06.383845 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:06.480337 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:06.480303 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:17.025280 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:17.025237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:17.850396 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:17.850320 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:24.221476 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:24.221437 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:24.321604 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:24.321569 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:24.429223 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:24.429194 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:24.524543 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:24.524511 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:24.622278 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:24.622253 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:25.412048 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:25.412010 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:25.511327 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:25.511300 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:25.611983 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:25.611945 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:26.501385 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:26.501338 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:46:27.279698 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:27.279643 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:33.994536 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:33.994484 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:34.060959 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:34.060926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:34.510560 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:34.510533 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:46:34.697729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:34.697692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:41.102948 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:41.102909 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:41.171471 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:41.171435 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:41.408830 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:41.408728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:41.676303 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:41.676214 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:41.973032 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:41.972846 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:42.297675 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:42.297640 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:46:44.926595 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:44.926541 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:45.022248 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:45.022199 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:45.103265 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:45.103237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:45.330294 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:45.330264 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:45.751436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:45.751402 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:45.857633 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:45.857599 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:49.427273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:49.427241 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:50.088273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:50.088241 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:50.183015 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:50.182988 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:50.534013 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:50.533986 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:50.637267 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:50.637234 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:46:57.111763 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:57.111728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:46:57.174440 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:46:57.174405 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:08.139942 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:08.139907 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:11.107926 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:11.107887 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:47:12.588005 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:12.587971 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:19.514321 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:19.514237 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:19.609863 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:19.609784 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:23.291768 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:23.291726 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:24.790939 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:24.790894 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:30.394191 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:30.394153 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:30.462095 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:30.462062 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:30.553894 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:30.553854 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:30.660214 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:30.660111 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:30.809737 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:30.809695 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:37.582110 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:37.582066 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:37.650624 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:37.650589 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:38.866736 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:38.866680 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:38.972711 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:38.972672 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:44.527840 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:44.527803 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:44.928279 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:44.928189 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:45.151359 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:45.151320 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:48.292550 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:48.292519 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:48.433574 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:48.433530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:48.530067 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:48.530043 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:49.238615 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:49.238585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:49.343529 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:49.343497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:49.454549 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:49.454526 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:52.825486 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:52.825446 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:47:55.792921 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:47:55.792884 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:48:24.050932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.050897 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf"] Jul 15 19:48:24.054245 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.054228 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.058729 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.058704 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73d57350\"/\"kube-root-ca.crt\"" Jul 15 19:48:24.058880 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.058748 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73d57350\"/\"default-dockercfg-p4wrf\"" Jul 15 19:48:24.058880 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.058818 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-73d57350\"/\"openshift-service-ca.crt\"" Jul 15 19:48:24.058880 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.058862 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-73d57350\"/\"e2e-lifecycle-model-73d57350-kserve-self-signed-certs\"" Jul 15 19:48:24.087180 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.087156 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf"] Jul 15 19:48:24.147997 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.147970 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:48:24.232127 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.232101 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-dshm\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.232248 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.232146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-model-cache\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.232248 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.232222 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eddb325-fad1-4eb3-9c16-dc62f992554a-tls-certs\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.232337 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.232252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-tmp-dir\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.232337 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.232271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cxss\" (UniqueName: \"kubernetes.io/projected/5eddb325-fad1-4eb3-9c16-dc62f992554a-kube-api-access-2cxss\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.232337 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.232292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-home\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333335 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eddb325-fad1-4eb3-9c16-dc62f992554a-tls-certs\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333335 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-tmp-dir\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333576 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cxss\" (UniqueName: \"kubernetes.io/projected/5eddb325-fad1-4eb3-9c16-dc62f992554a-kube-api-access-2cxss\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333576 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-home\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333576 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333480 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-dshm\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333741 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333584 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-model-cache\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333810 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-tmp-dir\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.333998 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.333975 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-home\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.334156 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.334109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-model-cache\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.335707 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.335690 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-dshm\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.335797 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.335779 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eddb325-fad1-4eb3-9c16-dc62f992554a-tls-certs\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.345387 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.345337 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cxss\" (UniqueName: \"kubernetes.io/projected/5eddb325-fad1-4eb3-9c16-dc62f992554a-kube-api-access-2cxss\") pod \"e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.363011 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.362990 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:24.505440 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:48:24.505392 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5eddb325_fad1_4eb3_9c16_dc62f992554a.slice/crio-6550d32d480252ee5e76fc80985a76f9ab6ed4462f86b1b33bbd499ef1bbf84b WatchSource:0}: Error finding container 6550d32d480252ee5e76fc80985a76f9ab6ed4462f86b1b33bbd499ef1bbf84b: Status 404 returned error can't find the container with id 6550d32d480252ee5e76fc80985a76f9ab6ed4462f86b1b33bbd499ef1bbf84b Jul 15 19:48:24.506603 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.506579 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf"] Jul 15 19:48:24.701621 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.701533 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" event={"ID":"5eddb325-fad1-4eb3-9c16-dc62f992554a","Type":"ContainerStarted","Data":"77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa"} Jul 15 19:48:24.701925 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.701850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" event={"ID":"5eddb325-fad1-4eb3-9c16-dc62f992554a","Type":"ContainerStarted","Data":"6550d32d480252ee5e76fc80985a76f9ab6ed4462f86b1b33bbd499ef1bbf84b"} Jul 15 19:48:24.737121 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:24.737076 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" podStartSLOduration=0.737062393 podStartE2EDuration="737.062393ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:48:24.731697718 +0000 UTC m=+2038.922575532" watchObservedRunningTime="2026-07-15 19:48:24.737062393 +0000 UTC m=+2038.927940198" Jul 15 19:48:27.455076 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:27.455045 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:48:34.363746 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:34.363705 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:34.364164 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:34.363787 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:34.376415 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:34.376390 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:34.758100 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:34.758074 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:48:43.756745 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:43.756712 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:48:58.117286 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:48:58.117202 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:01.617826 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:01.617789 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:49:02.150560 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:02.150523 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:07.155833 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:07.152640 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf"] Jul 15 19:49:07.267779 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:07.267746 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:22.175294 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.175237 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" podUID="5eddb325-fad1-4eb3-9c16-dc62f992554a" containerName="main" containerID="cri-o://77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa" gracePeriod=45 Jul 15 19:49:22.409970 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.409949 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:49:22.478798 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.478721 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-dshm\") pod \"5eddb325-fad1-4eb3-9c16-dc62f992554a\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " Jul 15 19:49:22.478798 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.478766 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-tmp-dir\") pod \"5eddb325-fad1-4eb3-9c16-dc62f992554a\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " Jul 15 19:49:22.479000 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.478799 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cxss\" (UniqueName: \"kubernetes.io/projected/5eddb325-fad1-4eb3-9c16-dc62f992554a-kube-api-access-2cxss\") pod \"5eddb325-fad1-4eb3-9c16-dc62f992554a\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " Jul 15 19:49:22.479000 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.478819 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-home\") pod \"5eddb325-fad1-4eb3-9c16-dc62f992554a\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " Jul 15 19:49:22.479000 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.478845 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-model-cache\") pod \"5eddb325-fad1-4eb3-9c16-dc62f992554a\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " Jul 15 19:49:22.479000 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.478889 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eddb325-fad1-4eb3-9c16-dc62f992554a-tls-certs\") pod \"5eddb325-fad1-4eb3-9c16-dc62f992554a\" (UID: \"5eddb325-fad1-4eb3-9c16-dc62f992554a\") " Jul 15 19:49:22.479215 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.479113 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-home" (OuterVolumeSpecName: "home") pod "5eddb325-fad1-4eb3-9c16-dc62f992554a" (UID: "5eddb325-fad1-4eb3-9c16-dc62f992554a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:49:22.479215 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.479114 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5eddb325-fad1-4eb3-9c16-dc62f992554a" (UID: "5eddb325-fad1-4eb3-9c16-dc62f992554a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:49:22.479215 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.479150 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-model-cache" (OuterVolumeSpecName: "model-cache") pod "5eddb325-fad1-4eb3-9c16-dc62f992554a" (UID: "5eddb325-fad1-4eb3-9c16-dc62f992554a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:49:22.480945 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.480918 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-dshm" (OuterVolumeSpecName: "dshm") pod "5eddb325-fad1-4eb3-9c16-dc62f992554a" (UID: "5eddb325-fad1-4eb3-9c16-dc62f992554a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:49:22.481060 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.480970 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eddb325-fad1-4eb3-9c16-dc62f992554a-kube-api-access-2cxss" (OuterVolumeSpecName: "kube-api-access-2cxss") pod "5eddb325-fad1-4eb3-9c16-dc62f992554a" (UID: "5eddb325-fad1-4eb3-9c16-dc62f992554a"). InnerVolumeSpecName "kube-api-access-2cxss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:49:22.481060 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.480993 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eddb325-fad1-4eb3-9c16-dc62f992554a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5eddb325-fad1-4eb3-9c16-dc62f992554a" (UID: "5eddb325-fad1-4eb3-9c16-dc62f992554a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:49:22.580286 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.580254 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:49:22.580427 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.580301 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:49:22.580427 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.580311 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cxss\" (UniqueName: \"kubernetes.io/projected/5eddb325-fad1-4eb3-9c16-dc62f992554a-kube-api-access-2cxss\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:49:22.580427 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.580321 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:49:22.580427 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.580332 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5eddb325-fad1-4eb3-9c16-dc62f992554a-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:49:22.580427 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.580339 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5eddb325-fad1-4eb3-9c16-dc62f992554a-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:49:22.978234 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.978201 2568 generic.go:356] "Generic (PLEG): container finished" podID="5eddb325-fad1-4eb3-9c16-dc62f992554a" containerID="77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa" exitCode=0 Jul 15 19:49:22.978378 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.978252 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" Jul 15 19:49:22.978378 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.978284 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" event={"ID":"5eddb325-fad1-4eb3-9c16-dc62f992554a","Type":"ContainerDied","Data":"77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa"} Jul 15 19:49:22.978378 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.978321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf" event={"ID":"5eddb325-fad1-4eb3-9c16-dc62f992554a","Type":"ContainerDied","Data":"6550d32d480252ee5e76fc80985a76f9ab6ed4462f86b1b33bbd499ef1bbf84b"} Jul 15 19:49:22.978378 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.978338 2568 scope.go:117] "RemoveContainer" containerID="77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa" Jul 15 19:49:22.987090 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.987075 2568 scope.go:117] "RemoveContainer" containerID="77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa" Jul 15 19:49:22.987326 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:49:22.987310 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa\": container with ID starting with 77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa not found: ID does not exist" containerID="77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa" Jul 15 19:49:22.987381 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:22.987334 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa"} err="failed to get container status \"77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa\": rpc error: code = NotFound desc = could not find container \"77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa\": container with ID starting with 77daf60229b4ba08635d51ed69fe68e48e7479c434c2bb8dfde577e21f87defa not found: ID does not exist" Jul 15 19:49:23.005187 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:23.005163 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf"] Jul 15 19:49:23.010263 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:23.010244 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-73d57350/e2e-lifecycle-model-73d57350-kserve-85698cc5cf-vzvxf"] Jul 15 19:49:24.227633 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:24.227601 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eddb325-fad1-4eb3-9c16-dc62f992554a" path="/var/lib/kubelet/pods/5eddb325-fad1-4eb3-9c16-dc62f992554a/volumes" Jul 15 19:49:43.350406 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:43.350370 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:46.980535 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:46.980499 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:49:53.241795 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:49:53.241741 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:10.257730 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:10.257693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:10.369810 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:10.369780 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:14.893239 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:14.893206 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:15.170846 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:15.170764 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:16.430251 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:16.430217 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:16.785394 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:16.785359 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:16.904357 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:16.904326 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:17.757699 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:17.757661 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:17.885962 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:17.885897 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:18.016077 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:18.016002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:19.430612 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:19.430580 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:50:19.612738 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:19.612698 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:21.459409 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:21.459376 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:28.227734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:28.227669 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:28.418461 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:28.418426 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:28.960020 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:28.959986 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:33.166961 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:33.166922 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:50:37.301258 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:37.301222 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:37.458391 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:37.458364 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:37.563630 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:37.563553 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:48.092981 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:48.092936 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:50:48.487721 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:50:48.487688 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:09.536086 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:09.536052 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:16.552642 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:16.552600 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:16.658474 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:16.658439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:16.985911 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:16.985871 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:17.462063 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:17.461969 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:17.583883 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:17.583853 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:20.508399 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:20.508367 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:20.615168 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:20.615134 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:20.815027 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:20.814952 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:20.932151 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:20.932117 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:21.548440 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:21.548396 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:21.650097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:21.650058 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:25.063584 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:25.063548 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:25.420203 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:25.420121 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:25.782611 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:25.782575 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:25.935810 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:25.935783 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:28.431694 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:28.431653 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:28.534793 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:28.534730 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:28.628115 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:28.628084 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:28.812575 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:28.812546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:29.228624 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:29.228533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:29.334831 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:29.334803 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:29.412608 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:29.412583 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:51:32.948568 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:32.948522 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:42.565470 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:42.565438 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:42.661934 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:42.661903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:43.807306 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:43.807273 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:51:50.148586 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:50.148501 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:50.569687 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:50.569654 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:51:52.880749 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:51:52.880710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:00.558706 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.558668 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2"] Jul 15 19:52:00.559103 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.559039 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eddb325-fad1-4eb3-9c16-dc62f992554a" containerName="main" Jul 15 19:52:00.559103 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.559056 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eddb325-fad1-4eb3-9c16-dc62f992554a" containerName="main" Jul 15 19:52:00.559184 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.559131 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eddb325-fad1-4eb3-9c16-dc62f992554a" containerName="main" Jul 15 19:52:00.562196 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.562173 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.566467 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.566442 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9130b1\"/\"openshift-service-ca.crt\"" Jul 15 19:52:00.567107 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.567091 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9130b1\"/\"kube-root-ca.crt\"" Jul 15 19:52:00.567209 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.567132 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9130b1\"/\"test-model-9130b1-kserve-self-signed-certs\"" Jul 15 19:52:00.567209 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.567154 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9130b1\"/\"default-dockercfg-95x8d\"" Jul 15 19:52:00.581663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.581643 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2"] Jul 15 19:52:00.646973 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.646944 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-home\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.646973 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.646975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-model-cache\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.647190 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.646998 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131864aa-7884-4f95-bf01-56e83294f77c-tls-certs\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.647190 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.647118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czgxc\" (UniqueName: \"kubernetes.io/projected/131864aa-7884-4f95-bf01-56e83294f77c-kube-api-access-czgxc\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.647190 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.647159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-tmp-dir\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.647190 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.647178 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-dshm\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.653572 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.653545 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:00.748097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-tmp-dir\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748097 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748098 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-dshm\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748310 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748221 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-home\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748310 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-model-cache\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748310 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131864aa-7884-4f95-bf01-56e83294f77c-tls-certs\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748510 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czgxc\" (UniqueName: \"kubernetes.io/projected/131864aa-7884-4f95-bf01-56e83294f77c-kube-api-access-czgxc\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748569 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748543 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-tmp-dir\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748632 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748609 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-home\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.748826 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.748808 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-model-cache\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.750297 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.750280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-dshm\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.750741 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.750723 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131864aa-7884-4f95-bf01-56e83294f77c-tls-certs\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.769746 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.769727 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czgxc\" (UniqueName: \"kubernetes.io/projected/131864aa-7884-4f95-bf01-56e83294f77c-kube-api-access-czgxc\") pod \"test-model-9130b1-kserve-544bc64bfc-l2mm2\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:00.873622 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:00.873545 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:01.020329 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:01.020300 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2"] Jul 15 19:52:01.020706 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:52:01.020687 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod131864aa_7884_4f95_bf01_56e83294f77c.slice/crio-0290ae5b24b948bd6e9645ff63b51d9beabfb98bb94d6a43319691fd5a11c674 WatchSource:0}: Error finding container 0290ae5b24b948bd6e9645ff63b51d9beabfb98bb94d6a43319691fd5a11c674: Status 404 returned error can't find the container with id 0290ae5b24b948bd6e9645ff63b51d9beabfb98bb94d6a43319691fd5a11c674 Jul 15 19:52:01.722364 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:01.722313 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" event={"ID":"131864aa-7884-4f95-bf01-56e83294f77c","Type":"ContainerStarted","Data":"feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088"} Jul 15 19:52:01.722783 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:01.722385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" event={"ID":"131864aa-7884-4f95-bf01-56e83294f77c","Type":"ContainerStarted","Data":"0290ae5b24b948bd6e9645ff63b51d9beabfb98bb94d6a43319691fd5a11c674"} Jul 15 19:52:01.756556 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:01.756503 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" podStartSLOduration=1.756489544 podStartE2EDuration="1.756489544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:52:01.755100745 +0000 UTC m=+2255.945978553" watchObservedRunningTime="2026-07-15 19:52:01.756489544 +0000 UTC m=+2255.947367350" Jul 15 19:52:03.724364 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:03.724310 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:52:08.773506 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:08.773474 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:08.836825 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:08.836794 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:10.874437 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:10.874395 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:10.874847 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:10.874459 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:10.886839 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:10.886812 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:11.783511 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:11.783484 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:52:15.616112 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:15.616081 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:52:39.015004 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.014970 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j"] Jul 15 19:52:39.018457 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.018437 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.021222 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.021204 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2908a6\"/\"openshift-service-ca.crt\"" Jul 15 19:52:39.021334 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.021237 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2908a6\"/\"default-dockercfg-h4ddn\"" Jul 15 19:52:39.022237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.022218 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2908a6\"/\"kube-root-ca.crt\"" Jul 15 19:52:39.022303 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.022240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2908a6\"/\"test-model-2908a6-kserve-self-signed-certs\"" Jul 15 19:52:39.035081 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.035057 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j"] Jul 15 19:52:39.046841 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.046801 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-tmp-dir\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.046993 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.046845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-dshm\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.046993 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.046874 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-home\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.046993 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.046898 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:39.046993 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.046970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f44b66-81ab-41de-a442-edf48913f3e4-tls-certs\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.047187 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.047013 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-model-cache\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.047187 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.047040 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z94td\" (UniqueName: \"kubernetes.io/projected/d2f44b66-81ab-41de-a442-edf48913f3e4-kube-api-access-z94td\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.147708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.147671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-dshm\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.147708 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.147713 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-home\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.147994 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.147756 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f44b66-81ab-41de-a442-edf48913f3e4-tls-certs\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.147994 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.147791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-model-cache\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.147994 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.147814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z94td\" (UniqueName: \"kubernetes.io/projected/d2f44b66-81ab-41de-a442-edf48913f3e4-kube-api-access-z94td\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.147994 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.147839 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-tmp-dir\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.148215 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.148160 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-home\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.148215 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.148200 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-model-cache\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.148304 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.148286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-tmp-dir\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.150035 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.150014 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-dshm\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.150265 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.150249 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f44b66-81ab-41de-a442-edf48913f3e4-tls-certs\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.158758 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.158733 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z94td\" (UniqueName: \"kubernetes.io/projected/d2f44b66-81ab-41de-a442-edf48913f3e4-kube-api-access-z94td\") pod \"test-model-2908a6-kserve-7f47789f5c-wdl7j\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.327061 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.326956 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:39.486892 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.486764 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j"] Jul 15 19:52:39.489390 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:52:39.489359 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2f44b66_81ab_41de_a442_edf48913f3e4.slice/crio-e6610c034c3d5157ba3326b59466fe3c91151b8f6933cea579214850030ab3c0 WatchSource:0}: Error finding container e6610c034c3d5157ba3326b59466fe3c91151b8f6933cea579214850030ab3c0: Status 404 returned error can't find the container with id e6610c034c3d5157ba3326b59466fe3c91151b8f6933cea579214850030ab3c0 Jul 15 19:52:39.908031 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.907996 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" event={"ID":"d2f44b66-81ab-41de-a442-edf48913f3e4","Type":"ContainerStarted","Data":"c8e4bd5a2fd7d1718f89254ececc7101b4ad73e478a80107a13de2bf0df0a376"} Jul 15 19:52:39.908031 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.908035 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" event={"ID":"d2f44b66-81ab-41de-a442-edf48913f3e4","Type":"ContainerStarted","Data":"e6610c034c3d5157ba3326b59466fe3c91151b8f6933cea579214850030ab3c0"} Jul 15 19:52:39.941115 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:39.941056 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" podStartSLOduration=1.941038647 podStartE2EDuration="1.941038647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:52:39.939644652 +0000 UTC m=+2294.130522459" watchObservedRunningTime="2026-07-15 19:52:39.941038647 +0000 UTC m=+2294.131916452" Jul 15 19:52:40.582284 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:40.582234 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:52:44.822609 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:44.822573 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:52:49.327301 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:49.327266 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:49.327711 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:49.327322 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:49.339992 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:49.339963 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:52:49.966070 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:52:49.966046 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:53:02.775067 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:02.775033 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2"] Jul 15 19:53:02.812819 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:02.812783 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:03.835887 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:03.835856 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j"] Jul 15 19:53:03.863786 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:03.863746 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:04.868759 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:04.868720 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:04.977923 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:04.977893 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:10.515185 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:10.515151 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:53:15.009054 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:15.009020 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:17.794418 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:17.794361 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" podUID="131864aa-7884-4f95-bf01-56e83294f77c" containerName="main" containerID="cri-o://feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088" gracePeriod=45 Jul 15 19:53:18.035638 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.035612 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:53:18.091399 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.091313 2568 generic.go:356] "Generic (PLEG): container finished" podID="131864aa-7884-4f95-bf01-56e83294f77c" containerID="feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088" exitCode=0 Jul 15 19:53:18.091399 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.091381 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" Jul 15 19:53:18.091571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.091404 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" event={"ID":"131864aa-7884-4f95-bf01-56e83294f77c","Type":"ContainerDied","Data":"feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088"} Jul 15 19:53:18.091571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.091447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2" event={"ID":"131864aa-7884-4f95-bf01-56e83294f77c","Type":"ContainerDied","Data":"0290ae5b24b948bd6e9645ff63b51d9beabfb98bb94d6a43319691fd5a11c674"} Jul 15 19:53:18.091571 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.091465 2568 scope.go:117] "RemoveContainer" containerID="feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088" Jul 15 19:53:18.099445 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.099430 2568 scope.go:117] "RemoveContainer" containerID="feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088" Jul 15 19:53:18.099703 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:53:18.099685 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088\": container with ID starting with feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088 not found: ID does not exist" containerID="feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088" Jul 15 19:53:18.099763 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.099712 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088"} err="failed to get container status \"feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088\": rpc error: code = NotFound desc = could not find container \"feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088\": container with ID starting with feaf115d527c00d7737f53dcb7e8f327227a5f338ca953d24d74af31572a3088 not found: ID does not exist" Jul 15 19:53:18.125470 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125450 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-tmp-dir\") pod \"131864aa-7884-4f95-bf01-56e83294f77c\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " Jul 15 19:53:18.125555 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125508 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czgxc\" (UniqueName: \"kubernetes.io/projected/131864aa-7884-4f95-bf01-56e83294f77c-kube-api-access-czgxc\") pod \"131864aa-7884-4f95-bf01-56e83294f77c\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " Jul 15 19:53:18.125555 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125538 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-model-cache\") pod \"131864aa-7884-4f95-bf01-56e83294f77c\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " Jul 15 19:53:18.125670 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125580 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131864aa-7884-4f95-bf01-56e83294f77c-tls-certs\") pod \"131864aa-7884-4f95-bf01-56e83294f77c\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " Jul 15 19:53:18.125670 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125606 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-home\") pod \"131864aa-7884-4f95-bf01-56e83294f77c\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " Jul 15 19:53:18.125670 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125627 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-dshm\") pod \"131864aa-7884-4f95-bf01-56e83294f77c\" (UID: \"131864aa-7884-4f95-bf01-56e83294f77c\") " Jul 15 19:53:18.125816 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125675 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "131864aa-7884-4f95-bf01-56e83294f77c" (UID: "131864aa-7884-4f95-bf01-56e83294f77c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:18.125874 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125856 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-model-cache" (OuterVolumeSpecName: "model-cache") pod "131864aa-7884-4f95-bf01-56e83294f77c" (UID: "131864aa-7884-4f95-bf01-56e83294f77c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:18.125934 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125874 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:18.125934 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.125906 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-home" (OuterVolumeSpecName: "home") pod "131864aa-7884-4f95-bf01-56e83294f77c" (UID: "131864aa-7884-4f95-bf01-56e83294f77c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:18.127674 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.127652 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-dshm" (OuterVolumeSpecName: "dshm") pod "131864aa-7884-4f95-bf01-56e83294f77c" (UID: "131864aa-7884-4f95-bf01-56e83294f77c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:18.127784 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.127742 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/131864aa-7884-4f95-bf01-56e83294f77c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "131864aa-7884-4f95-bf01-56e83294f77c" (UID: "131864aa-7884-4f95-bf01-56e83294f77c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:53:18.127784 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.127762 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/131864aa-7884-4f95-bf01-56e83294f77c-kube-api-access-czgxc" (OuterVolumeSpecName: "kube-api-access-czgxc") pod "131864aa-7884-4f95-bf01-56e83294f77c" (UID: "131864aa-7884-4f95-bf01-56e83294f77c"). InnerVolumeSpecName "kube-api-access-czgxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:53:18.226189 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.226169 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:18.226189 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.226188 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/131864aa-7884-4f95-bf01-56e83294f77c-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:18.226311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.226196 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:18.226311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.226203 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/131864aa-7884-4f95-bf01-56e83294f77c-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:18.226311 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.226211 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czgxc\" (UniqueName: \"kubernetes.io/projected/131864aa-7884-4f95-bf01-56e83294f77c-kube-api-access-czgxc\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:18.411182 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.411127 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2"] Jul 15 19:53:18.420531 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.420509 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9130b1/test-model-9130b1-kserve-544bc64bfc-l2mm2"] Jul 15 19:53:18.855746 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:18.855709 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" podUID="d2f44b66-81ab-41de-a442-edf48913f3e4" containerName="main" containerID="cri-o://c8e4bd5a2fd7d1718f89254ececc7101b4ad73e478a80107a13de2bf0df0a376" gracePeriod=45 Jul 15 19:53:19.099210 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.099186 2568 generic.go:356] "Generic (PLEG): container finished" podID="d2f44b66-81ab-41de-a442-edf48913f3e4" containerID="c8e4bd5a2fd7d1718f89254ececc7101b4ad73e478a80107a13de2bf0df0a376" exitCode=0 Jul 15 19:53:19.099339 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.099223 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" event={"ID":"d2f44b66-81ab-41de-a442-edf48913f3e4","Type":"ContainerDied","Data":"c8e4bd5a2fd7d1718f89254ececc7101b4ad73e478a80107a13de2bf0df0a376"} Jul 15 19:53:19.099339 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.099248 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" event={"ID":"d2f44b66-81ab-41de-a442-edf48913f3e4","Type":"ContainerDied","Data":"e6610c034c3d5157ba3326b59466fe3c91151b8f6933cea579214850030ab3c0"} Jul 15 19:53:19.099339 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.099261 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6610c034c3d5157ba3326b59466fe3c91151b8f6933cea579214850030ab3c0" Jul 15 19:53:19.111282 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.111230 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:53:19.145305 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.145283 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:19.231404 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231378 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z94td\" (UniqueName: \"kubernetes.io/projected/d2f44b66-81ab-41de-a442-edf48913f3e4-kube-api-access-z94td\") pod \"d2f44b66-81ab-41de-a442-edf48913f3e4\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " Jul 15 19:53:19.231507 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231431 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-dshm\") pod \"d2f44b66-81ab-41de-a442-edf48913f3e4\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " Jul 15 19:53:19.231507 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231492 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-home\") pod \"d2f44b66-81ab-41de-a442-edf48913f3e4\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " Jul 15 19:53:19.231625 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231527 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-tmp-dir\") pod \"d2f44b66-81ab-41de-a442-edf48913f3e4\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " Jul 15 19:53:19.231625 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231559 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f44b66-81ab-41de-a442-edf48913f3e4-tls-certs\") pod \"d2f44b66-81ab-41de-a442-edf48913f3e4\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " Jul 15 19:53:19.231625 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231618 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-model-cache\") pod \"d2f44b66-81ab-41de-a442-edf48913f3e4\" (UID: \"d2f44b66-81ab-41de-a442-edf48913f3e4\") " Jul 15 19:53:19.231805 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231778 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-home" (OuterVolumeSpecName: "home") pod "d2f44b66-81ab-41de-a442-edf48913f3e4" (UID: "d2f44b66-81ab-41de-a442-edf48913f3e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:19.231857 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231808 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2f44b66-81ab-41de-a442-edf48913f3e4" (UID: "d2f44b66-81ab-41de-a442-edf48913f3e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:19.231932 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.231904 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2f44b66-81ab-41de-a442-edf48913f3e4" (UID: "d2f44b66-81ab-41de-a442-edf48913f3e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:19.233664 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.233635 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f44b66-81ab-41de-a442-edf48913f3e4-kube-api-access-z94td" (OuterVolumeSpecName: "kube-api-access-z94td") pod "d2f44b66-81ab-41de-a442-edf48913f3e4" (UID: "d2f44b66-81ab-41de-a442-edf48913f3e4"). InnerVolumeSpecName "kube-api-access-z94td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:53:19.233757 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.233685 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-dshm" (OuterVolumeSpecName: "dshm") pod "d2f44b66-81ab-41de-a442-edf48913f3e4" (UID: "d2f44b66-81ab-41de-a442-edf48913f3e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:53:19.233757 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.233701 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f44b66-81ab-41de-a442-edf48913f3e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2f44b66-81ab-41de-a442-edf48913f3e4" (UID: "d2f44b66-81ab-41de-a442-edf48913f3e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:53:19.332856 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.332836 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:19.332856 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.332856 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f44b66-81ab-41de-a442-edf48913f3e4-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:19.332979 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.332865 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:19.332979 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.332873 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z94td\" (UniqueName: \"kubernetes.io/projected/d2f44b66-81ab-41de-a442-edf48913f3e4-kube-api-access-z94td\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:19.332979 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.332881 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:19.332979 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:19.332889 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f44b66-81ab-41de-a442-edf48913f3e4-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:53:20.104036 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:20.104010 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j" Jul 15 19:53:20.137314 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:20.137288 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j"] Jul 15 19:53:20.142057 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:20.142031 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2908a6/test-model-2908a6-kserve-7f47789f5c-wdl7j"] Jul 15 19:53:20.228473 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:20.228444 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="131864aa-7884-4f95-bf01-56e83294f77c" path="/var/lib/kubelet/pods/131864aa-7884-4f95-bf01-56e83294f77c/volumes" Jul 15 19:53:20.228832 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:20.228819 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f44b66-81ab-41de-a442-edf48913f3e4" path="/var/lib/kubelet/pods/d2f44b66-81ab-41de-a442-edf48913f3e4/volumes" Jul 15 19:53:21.406463 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:21.406433 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:53:26.277242 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:26.277205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:26.383870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:26.383837 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:29.944745 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:29.944705 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:30.173990 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:30.173955 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:30.981043 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:30.981002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:31.082040 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:31.082009 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:41.122755 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:41.122711 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:43.558192 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:43.558150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:43.634183 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:43.634150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:44.265640 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:44.265612 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:53:45.472026 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:45.471991 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:52.786309 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:52.786276 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:52.862715 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:52.862691 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:56.358171 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:56.358133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:56.604201 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:56.604161 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:56.967368 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:56.967328 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:57.415386 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:57.415330 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:53:57.589087 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:53:57.589052 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:01.278199 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:01.278167 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:01.368054 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:01.368019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:01.555211 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:01.555130 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:03.109921 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:03.109888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:03.207508 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:03.207479 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:05.804640 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:05.804607 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:06.300608 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:06.300574 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:06.802998 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:06.802960 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:07.112337 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:07.112236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:09.470412 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:09.470383 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:09.537449 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:09.537419 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:54:09.579351 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:09.579320 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:09.690909 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:09.690866 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:09.957233 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:09.957196 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:10.558452 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:10.558408 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:10.664090 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:10.664039 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:13.930365 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:13.930303 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:22.281889 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.281854 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs"] Jul 15 19:54:22.282273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.282148 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f44b66-81ab-41de-a442-edf48913f3e4" containerName="main" Jul 15 19:54:22.282273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.282161 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f44b66-81ab-41de-a442-edf48913f3e4" containerName="main" Jul 15 19:54:22.282273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.282182 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="131864aa-7884-4f95-bf01-56e83294f77c" containerName="main" Jul 15 19:54:22.282273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.282189 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="131864aa-7884-4f95-bf01-56e83294f77c" containerName="main" Jul 15 19:54:22.282273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.282254 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="131864aa-7884-4f95-bf01-56e83294f77c" containerName="main" Jul 15 19:54:22.282273 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.282263 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f44b66-81ab-41de-a442-edf48913f3e4" containerName="main" Jul 15 19:54:22.285226 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.285210 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" Jul 15 19:54:22.290532 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.290509 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 19:54:22.290532 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.290509 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhrw8\"" Jul 15 19:54:22.303499 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.303475 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs"] Jul 15 19:54:22.337565 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.337544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb38451-2c44-4153-9ff7-e8e2c98d3846-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs\" (UID: \"3cb38451-2c44-4153-9ff7-e8e2c98d3846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" Jul 15 19:54:22.438423 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.438393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb38451-2c44-4153-9ff7-e8e2c98d3846-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs\" (UID: \"3cb38451-2c44-4153-9ff7-e8e2c98d3846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" Jul 15 19:54:22.438920 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.438898 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb38451-2c44-4153-9ff7-e8e2c98d3846-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs\" (UID: \"3cb38451-2c44-4153-9ff7-e8e2c98d3846\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" Jul 15 19:54:22.595267 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.595204 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" Jul 15 19:54:22.733620 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.733585 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs"] Jul 15 19:54:22.735869 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:54:22.735837 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb38451_2c44_4153_9ff7_e8e2c98d3846.slice/crio-ecaed372e0c2973cb2fe55c8b4f61632414c4e48c59e3454fa7a362a32ee2738 WatchSource:0}: Error finding container ecaed372e0c2973cb2fe55c8b4f61632414c4e48c59e3454fa7a362a32ee2738: Status 404 returned error can't find the container with id ecaed372e0c2973cb2fe55c8b4f61632414c4e48c59e3454fa7a362a32ee2738 Jul 15 19:54:22.737837 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:22.737816 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 19:54:23.134610 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:23.134560 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:23.234576 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:23.234527 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:23.405912 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:23.405818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" event={"ID":"3cb38451-2c44-4153-9ff7-e8e2c98d3846","Type":"ContainerStarted","Data":"ecaed372e0c2973cb2fe55c8b4f61632414c4e48c59e3454fa7a362a32ee2738"} Jul 15 19:54:26.952583 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:26.952555 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:54:31.803317 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:31.803278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:31.907436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:31.907402 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:39.482313 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:39.482220 2568 generic.go:356] "Generic (PLEG): container finished" podID="3cb38451-2c44-4153-9ff7-e8e2c98d3846" containerID="00b4ac09612b90d75f230011a78d8d5dfb120bc12e33659fb4229005c75350a9" exitCode=0 Jul 15 19:54:39.482313 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:39.482266 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" event={"ID":"3cb38451-2c44-4153-9ff7-e8e2c98d3846","Type":"ContainerDied","Data":"00b4ac09612b90d75f230011a78d8d5dfb120bc12e33659fb4229005c75350a9"} Jul 15 19:54:40.615242 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:40.615218 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" Jul 15 19:54:40.790718 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:40.790643 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb38451-2c44-4153-9ff7-e8e2c98d3846-maas-api-service-ca\") pod \"3cb38451-2c44-4153-9ff7-e8e2c98d3846\" (UID: \"3cb38451-2c44-4153-9ff7-e8e2c98d3846\") " Jul 15 19:54:40.790974 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:40.790952 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb38451-2c44-4153-9ff7-e8e2c98d3846-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cb38451-2c44-4153-9ff7-e8e2c98d3846" (UID: "3cb38451-2c44-4153-9ff7-e8e2c98d3846"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 19:54:40.891886 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:40.891856 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cb38451-2c44-4153-9ff7-e8e2c98d3846-maas-api-service-ca\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:54:41.493182 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:41.493154 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" Jul 15 19:54:41.493376 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:41.493161 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs" event={"ID":"3cb38451-2c44-4153-9ff7-e8e2c98d3846","Type":"ContainerDied","Data":"ecaed372e0c2973cb2fe55c8b4f61632414c4e48c59e3454fa7a362a32ee2738"} Jul 15 19:54:41.493376 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:41.493280 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecaed372e0c2973cb2fe55c8b4f61632414c4e48c59e3454fa7a362a32ee2738" Jul 15 19:54:43.279502 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:43.279469 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:43.399286 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:43.399255 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:43.764106 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:43.764075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:47.241268 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:47.241235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:54:47.448180 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:47.448147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:55.108224 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:55.108140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:55.209920 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:55.209885 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:58.402410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:58.402336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:54:58.629851 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:54:58.629798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:00.312410 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:00.312333 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:00.433991 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:00.433952 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:09.552357 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:09.552316 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:12.614663 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:12.614630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:12.637024 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:12.636934 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:55:12.714442 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:12.714405 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:13.963004 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:13.962963 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:22.011756 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:22.011722 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:22.110616 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:22.110583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:24.807068 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:24.807028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:25.056734 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:25.056702 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:25.479162 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:25.479126 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:26.618767 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:26.618733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:26.722306 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:26.722258 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:29.145620 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:29.145586 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:29.243662 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:29.243627 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:29.437360 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:29.437254 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:30.151775 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:30.151743 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:30.235058 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:30.235030 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:33.437415 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:33.437375 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:33.801085 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:33.801050 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:33.937997 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:33.937966 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:55:34.335566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:34.335529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:34.591232 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:34.591139 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:38.128067 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:38.128027 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:38.206256 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:38.206223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:38.550539 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:38.550499 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:38.881869 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:38.881775 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:39.574176 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:39.574140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:39.705706 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:39.705669 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:42.546163 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:42.546111 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:48.815048 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:48.815015 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:55:51.477293 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:51.477248 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:55:51.559662 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:55:51.559629 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:01.897267 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:01.897232 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:01.958147 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:01.958107 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:17.480532 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:17.480497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:20.386554 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.386473 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf"] Jul 15 19:56:20.386919 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.386768 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb38451-2c44-4153-9ff7-e8e2c98d3846" containerName="revoke-keys" Jul 15 19:56:20.386919 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.386778 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb38451-2c44-4153-9ff7-e8e2c98d3846" containerName="revoke-keys" Jul 15 19:56:20.386919 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.386853 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb38451-2c44-4153-9ff7-e8e2c98d3846" containerName="revoke-keys" Jul 15 19:56:20.389697 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.389681 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.397701 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.397680 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"auth-test-model-77bf75-kserve-self-signed-certs\"" Jul 15 19:56:20.398435 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.398419 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"openshift-service-ca.crt\"" Jul 15 19:56:20.398503 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.398488 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"default-dockercfg-7kdrr\"" Jul 15 19:56:20.398555 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.398421 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"kube-root-ca.crt\"" Jul 15 19:56:20.425120 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.425093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf"] Jul 15 19:56:20.438044 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.438015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:20.493244 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.493209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tmp-dir\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.493441 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.493264 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-model-cache\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.493441 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.493312 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-home\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.493441 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.493360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxpz\" (UniqueName: \"kubernetes.io/projected/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-kube-api-access-tnxpz\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.493441 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.493419 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-dshm\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.493594 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.493453 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tls-certs\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.594494 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.594452 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-home\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.594602 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.594506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxpz\" (UniqueName: \"kubernetes.io/projected/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-kube-api-access-tnxpz\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.594643 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.594622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-dshm\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.594679 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.594661 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tls-certs\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.594757 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.594741 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tmp-dir\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.594837 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.594821 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-model-cache\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.594897 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.594850 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-home\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.595042 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.595023 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tmp-dir\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.595139 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.595122 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-model-cache\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.596877 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.596851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-dshm\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.597087 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.597071 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tls-certs\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.607794 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.607771 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxpz\" (UniqueName: \"kubernetes.io/projected/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-kube-api-access-tnxpz\") pod \"auth-test-model-77bf75-kserve-96d99795f-7ttmf\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.698387 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.698282 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:20.847196 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.847168 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf"] Jul 15 19:56:20.849539 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:56:20.849513 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeff8a0ac_4c5b_444b_a869_8ed29eaa9b46.slice/crio-fd8be962c74aa06ac0d5f73719ed1f1159be4fd4dccd6cc9a051d1128960b140 WatchSource:0}: Error finding container fd8be962c74aa06ac0d5f73719ed1f1159be4fd4dccd6cc9a051d1128960b140: Status 404 returned error can't find the container with id fd8be962c74aa06ac0d5f73719ed1f1159be4fd4dccd6cc9a051d1128960b140 Jul 15 19:56:20.969899 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.969810 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" event={"ID":"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46","Type":"ContainerStarted","Data":"17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5"} Jul 15 19:56:20.969899 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.969854 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" event={"ID":"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46","Type":"ContainerStarted","Data":"fd8be962c74aa06ac0d5f73719ed1f1159be4fd4dccd6cc9a051d1128960b140"} Jul 15 19:56:20.999280 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:20.999226 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" podStartSLOduration=0.999208678 podStartE2EDuration="999.208678ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:56:20.997730693 +0000 UTC m=+2515.188608498" watchObservedRunningTime="2026-07-15 19:56:20.999208678 +0000 UTC m=+2515.190086483" Jul 15 19:56:23.753728 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:23.753689 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:56:30.699315 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:30.699277 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:30.699714 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:30.699333 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:30.711894 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:30.711871 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:31.027871 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:31.027847 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:56:34.581417 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:34.581370 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:40.055706 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.055675 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865"] Jul 15 19:56:40.059044 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.059023 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.062853 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.062829 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"default-dockercfg-djcbj\"" Jul 15 19:56:40.062973 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.062877 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"kube-root-ca.crt\"" Jul 15 19:56:40.062973 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.062892 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"openshift-service-ca.crt\"" Jul 15 19:56:40.062973 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.062944 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"auth-test-model-fcd095-kserve-self-signed-certs\"" Jul 15 19:56:40.072768 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.072747 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:56:40.094493 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.094472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865"] Jul 15 19:56:40.155696 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.155668 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492305f0-45da-48f3-b3bb-9451e651cd67-tls-certs\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.155836 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.155702 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjk7\" (UniqueName: \"kubernetes.io/projected/492305f0-45da-48f3-b3bb-9451e651cd67-kube-api-access-8hjk7\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.155836 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.155737 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-dshm\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.155836 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.155760 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-model-cache\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.156001 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.155875 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-home\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.156001 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.155954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-tmp-dir\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256521 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256498 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-home\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-tmp-dir\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256554 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492305f0-45da-48f3-b3bb-9451e651cd67-tls-certs\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjk7\" (UniqueName: \"kubernetes.io/projected/492305f0-45da-48f3-b3bb-9451e651cd67-kube-api-access-8hjk7\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-dshm\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-model-cache\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256940 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256865 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-home\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256941 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-tmp-dir\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.256987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.256961 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-model-cache\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.258801 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.258773 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-dshm\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.259035 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.259018 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492305f0-45da-48f3-b3bb-9451e651cd67-tls-certs\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.267026 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.267002 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjk7\" (UniqueName: \"kubernetes.io/projected/492305f0-45da-48f3-b3bb-9451e651cd67-kube-api-access-8hjk7\") pod \"auth-test-model-fcd095-kserve-7cff7b8c47-cp865\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.367992 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.367935 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:40.503891 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:40.503869 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865"] Jul 15 19:56:40.504750 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:56:40.504728 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod492305f0_45da_48f3_b3bb_9451e651cd67.slice/crio-6ca930f178bdeee7e2e9e0470b513385ce7323bda646214b4b419ed2bbdc5892 WatchSource:0}: Error finding container 6ca930f178bdeee7e2e9e0470b513385ce7323bda646214b4b419ed2bbdc5892: Status 404 returned error can't find the container with id 6ca930f178bdeee7e2e9e0470b513385ce7323bda646214b4b419ed2bbdc5892 Jul 15 19:56:41.066964 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:41.066931 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" event={"ID":"492305f0-45da-48f3-b3bb-9451e651cd67","Type":"ContainerStarted","Data":"0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f"} Jul 15 19:56:41.067332 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:41.066972 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" event={"ID":"492305f0-45da-48f3-b3bb-9451e651cd67","Type":"ContainerStarted","Data":"6ca930f178bdeee7e2e9e0470b513385ce7323bda646214b4b419ed2bbdc5892"} Jul 15 19:56:41.099427 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:41.099383 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" podStartSLOduration=1.099368147 podStartE2EDuration="1.099368147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:56:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:56:41.098082699 +0000 UTC m=+2535.288960516" watchObservedRunningTime="2026-07-15 19:56:41.099368147 +0000 UTC m=+2535.290245986" Jul 15 19:56:43.823149 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:43.823119 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:56:50.368394 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:50.368339 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:50.368797 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:50.368417 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:50.380968 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:50.380940 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:51.127933 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:51.127903 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:56:57.720321 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:56:57.720291 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:57:17.628095 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:57:17.628060 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:57:33.708209 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:57:33.708175 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:57:41.336618 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:57:41.336581 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:57:43.606219 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:57:43.606180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:57:51.614304 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:57:51.614227 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:58:12.415405 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:58:12.415370 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:58:27.607438 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:58:27.607408 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:58:45.045679 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:58:45.045646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:58:56.170960 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:58:56.170927 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf"] Jul 15 19:58:56.180017 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:58:56.179985 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:58:57.122594 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:58:57.122563 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865"] Jul 15 19:58:57.150066 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:58:57.150040 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:00.528620 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:00.528583 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:59:08.093211 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.093178 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49"] Jul 15 19:59:08.097555 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.097517 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.113070 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.113052 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"sub-test-model-77bf75-kserve-self-signed-certs\"" Jul 15 19:59:08.117831 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.117806 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:08.130392 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.130371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867a345b-fa3a-487b-8c36-bc91bd099495-tls-certs\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.130484 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.130425 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9djng\" (UniqueName: \"kubernetes.io/projected/867a345b-fa3a-487b-8c36-bc91bd099495-kube-api-access-9djng\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.130484 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.130466 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-model-cache\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.130562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.130516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-dshm\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.130562 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.130541 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-tmp-dir\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.130628 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.130562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-home\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.141972 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.141940 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49"] Jul 15 19:59:08.231047 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867a345b-fa3a-487b-8c36-bc91bd099495-tls-certs\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231231 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231071 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9djng\" (UniqueName: \"kubernetes.io/projected/867a345b-fa3a-487b-8c36-bc91bd099495-kube-api-access-9djng\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231231 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231093 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-model-cache\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231231 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231120 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-dshm\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231231 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-tmp-dir\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231231 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-home\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231593 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231569 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-model-cache\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231592 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-home\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.231657 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.231634 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-tmp-dir\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.233337 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.233316 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-dshm\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.233635 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.233615 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867a345b-fa3a-487b-8c36-bc91bd099495-tls-certs\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.243588 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.243569 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9djng\" (UniqueName: \"kubernetes.io/projected/867a345b-fa3a-487b-8c36-bc91bd099495-kube-api-access-9djng\") pod \"sub-test-model-77bf75-kserve-585868c995-5lz49\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.408033 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.407952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:08.567756 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.567727 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49"] Jul 15 19:59:08.568405 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:59:08.568377 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod867a345b_fa3a_487b_8c36_bc91bd099495.slice/crio-74617eb8dd79e0f4c100440b03771e4f74457764bf3a42c1c940a1493bb7a412 WatchSource:0}: Error finding container 74617eb8dd79e0f4c100440b03771e4f74457764bf3a42c1c940a1493bb7a412: Status 404 returned error can't find the container with id 74617eb8dd79e0f4c100440b03771e4f74457764bf3a42c1c940a1493bb7a412 Jul 15 19:59:08.774987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.774953 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" event={"ID":"867a345b-fa3a-487b-8c36-bc91bd099495","Type":"ContainerStarted","Data":"067b2af74c89f5ce104456af2e96c7e0c6f797db79d35fffbba5e86c71378f73"} Jul 15 19:59:08.774987 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.774998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" event={"ID":"867a345b-fa3a-487b-8c36-bc91bd099495","Type":"ContainerStarted","Data":"74617eb8dd79e0f4c100440b03771e4f74457764bf3a42c1c940a1493bb7a412"} Jul 15 19:59:08.805684 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:08.805630 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" podStartSLOduration=0.805613378 podStartE2EDuration="805.613378ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:08.803698889 +0000 UTC m=+2682.994576696" watchObservedRunningTime="2026-07-15 19:59:08.805613378 +0000 UTC m=+2682.996491258" Jul 15 19:59:11.190683 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.190626 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" podUID="eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" containerName="main" containerID="cri-o://17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5" gracePeriod=45 Jul 15 19:59:11.439330 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.439304 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:59:11.561456 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561422 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tmp-dir\") pod \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " Jul 15 19:59:11.561456 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561459 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnxpz\" (UniqueName: \"kubernetes.io/projected/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-kube-api-access-tnxpz\") pod \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " Jul 15 19:59:11.561677 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561510 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tls-certs\") pod \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " Jul 15 19:59:11.561677 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561551 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-model-cache\") pod \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " Jul 15 19:59:11.561677 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561584 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-dshm\") pod \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " Jul 15 19:59:11.561677 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561647 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-home\") pod \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\" (UID: \"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46\") " Jul 15 19:59:11.561879 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561746 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" (UID: "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:11.561879 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561846 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-model-cache" (OuterVolumeSpecName: "model-cache") pod "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" (UID: "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:11.561992 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561936 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-home" (OuterVolumeSpecName: "home") pod "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" (UID: "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:11.561992 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561946 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:11.561992 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.561987 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:11.563870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.563842 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-dshm" (OuterVolumeSpecName: "dshm") pod "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" (UID: "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:11.563870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.563861 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" (UID: "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:59:11.564015 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.563931 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-kube-api-access-tnxpz" (OuterVolumeSpecName: "kube-api-access-tnxpz") pod "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" (UID: "eff8a0ac-4c5b-444b-a869-8ed29eaa9b46"). InnerVolumeSpecName "kube-api-access-tnxpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:59:11.663355 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.663324 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:11.663509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.663366 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:11.663509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.663379 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:11.663509 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.663387 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnxpz\" (UniqueName: \"kubernetes.io/projected/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46-kube-api-access-tnxpz\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:11.790705 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.790677 2568 generic.go:356] "Generic (PLEG): container finished" podID="eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" containerID="17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5" exitCode=0 Jul 15 19:59:11.790870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.790720 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" Jul 15 19:59:11.790870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.790772 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" event={"ID":"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46","Type":"ContainerDied","Data":"17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5"} Jul 15 19:59:11.790870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.790818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf" event={"ID":"eff8a0ac-4c5b-444b-a869-8ed29eaa9b46","Type":"ContainerDied","Data":"fd8be962c74aa06ac0d5f73719ed1f1159be4fd4dccd6cc9a051d1128960b140"} Jul 15 19:59:11.790870 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.790839 2568 scope.go:117] "RemoveContainer" containerID="17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5" Jul 15 19:59:11.799808 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.799789 2568 scope.go:117] "RemoveContainer" containerID="17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5" Jul 15 19:59:11.800091 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:59:11.800073 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5\": container with ID starting with 17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5 not found: ID does not exist" containerID="17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5" Jul 15 19:59:11.800172 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.800102 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5"} err="failed to get container status \"17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5\": rpc error: code = NotFound desc = could not find container \"17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5\": container with ID starting with 17fc6e5951e3ee0f7d1c97450d91bec62ab165553cab233c9703d668454451a5 not found: ID does not exist" Jul 15 19:59:11.821287 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.821235 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf"] Jul 15 19:59:11.826141 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:11.826119 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/auth-test-model-77bf75-kserve-96d99795f-7ttmf"] Jul 15 19:59:12.143611 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.143514 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" podUID="492305f0-45da-48f3-b3bb-9451e651cd67" containerName="main" containerID="cri-o://0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f" gracePeriod=45 Jul 15 19:59:12.228388 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.228362 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" path="/var/lib/kubelet/pods/eff8a0ac-4c5b-444b-a869-8ed29eaa9b46/volumes" Jul 15 19:59:12.390028 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.390003 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:59:12.468877 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.468785 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492305f0-45da-48f3-b3bb-9451e651cd67-tls-certs\") pod \"492305f0-45da-48f3-b3bb-9451e651cd67\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " Jul 15 19:59:12.468877 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.468827 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-model-cache\") pod \"492305f0-45da-48f3-b3bb-9451e651cd67\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " Jul 15 19:59:12.469115 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.468910 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hjk7\" (UniqueName: \"kubernetes.io/projected/492305f0-45da-48f3-b3bb-9451e651cd67-kube-api-access-8hjk7\") pod \"492305f0-45da-48f3-b3bb-9451e651cd67\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " Jul 15 19:59:12.469115 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.468933 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-tmp-dir\") pod \"492305f0-45da-48f3-b3bb-9451e651cd67\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " Jul 15 19:59:12.469115 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.468958 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-home\") pod \"492305f0-45da-48f3-b3bb-9451e651cd67\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " Jul 15 19:59:12.469115 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.468996 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-dshm\") pod \"492305f0-45da-48f3-b3bb-9451e651cd67\" (UID: \"492305f0-45da-48f3-b3bb-9451e651cd67\") " Jul 15 19:59:12.469300 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.469172 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-model-cache" (OuterVolumeSpecName: "model-cache") pod "492305f0-45da-48f3-b3bb-9451e651cd67" (UID: "492305f0-45da-48f3-b3bb-9451e651cd67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:12.469300 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.469188 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "492305f0-45da-48f3-b3bb-9451e651cd67" (UID: "492305f0-45da-48f3-b3bb-9451e651cd67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:12.469300 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.469258 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-home" (OuterVolumeSpecName: "home") pod "492305f0-45da-48f3-b3bb-9451e651cd67" (UID: "492305f0-45da-48f3-b3bb-9451e651cd67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:12.471056 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.471026 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-dshm" (OuterVolumeSpecName: "dshm") pod "492305f0-45da-48f3-b3bb-9451e651cd67" (UID: "492305f0-45da-48f3-b3bb-9451e651cd67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 19:59:12.471178 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.471103 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492305f0-45da-48f3-b3bb-9451e651cd67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "492305f0-45da-48f3-b3bb-9451e651cd67" (UID: "492305f0-45da-48f3-b3bb-9451e651cd67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 19:59:12.471178 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.471110 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492305f0-45da-48f3-b3bb-9451e651cd67-kube-api-access-8hjk7" (OuterVolumeSpecName: "kube-api-access-8hjk7") pod "492305f0-45da-48f3-b3bb-9451e651cd67" (UID: "492305f0-45da-48f3-b3bb-9451e651cd67"). InnerVolumeSpecName "kube-api-access-8hjk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 19:59:12.569795 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.569759 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:12.569795 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.569787 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjk7\" (UniqueName: \"kubernetes.io/projected/492305f0-45da-48f3-b3bb-9451e651cd67-kube-api-access-8hjk7\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:12.569795 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.569797 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:12.570070 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.569806 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:12.570070 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.569813 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/492305f0-45da-48f3-b3bb-9451e651cd67-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:12.570070 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.569821 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/492305f0-45da-48f3-b3bb-9451e651cd67-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 19:59:12.797020 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.796984 2568 generic.go:356] "Generic (PLEG): container finished" podID="492305f0-45da-48f3-b3bb-9451e651cd67" containerID="0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f" exitCode=0 Jul 15 19:59:12.797201 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.797046 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" Jul 15 19:59:12.797201 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.797073 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" event={"ID":"492305f0-45da-48f3-b3bb-9451e651cd67","Type":"ContainerDied","Data":"0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f"} Jul 15 19:59:12.797201 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.797113 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865" event={"ID":"492305f0-45da-48f3-b3bb-9451e651cd67","Type":"ContainerDied","Data":"6ca930f178bdeee7e2e9e0470b513385ce7323bda646214b4b419ed2bbdc5892"} Jul 15 19:59:12.797201 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.797132 2568 scope.go:117] "RemoveContainer" containerID="0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f" Jul 15 19:59:12.806292 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.806276 2568 scope.go:117] "RemoveContainer" containerID="0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f" Jul 15 19:59:12.806598 ip-10-0-132-25 kubenswrapper[2568]: E0715 19:59:12.806579 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f\": container with ID starting with 0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f not found: ID does not exist" containerID="0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f" Jul 15 19:59:12.806676 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.806607 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f"} err="failed to get container status \"0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f\": rpc error: code = NotFound desc = could not find container \"0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f\": container with ID starting with 0ae740b47803976da8246e26548f4b87e9c3723a854302c2cc2d1e16937fc38f not found: ID does not exist" Jul 15 19:59:12.825651 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.825619 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865"] Jul 15 19:59:12.835397 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:12.835369 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/auth-test-model-fcd095-kserve-7cff7b8c47-cp865"] Jul 15 19:59:14.228527 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:14.228488 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="492305f0-45da-48f3-b3bb-9451e651cd67" path="/var/lib/kubelet/pods/492305f0-45da-48f3-b3bb-9451e651cd67/volumes" Jul 15 19:59:18.408582 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:18.408490 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:18.408582 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:18.408545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:18.420905 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:18.420879 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:18.838656 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:18.838626 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 19:59:21.726445 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:21.726407 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:59:27.946263 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:27.946231 2568 scope.go:117] "RemoveContainer" containerID="c8e4bd5a2fd7d1718f89254ececc7101b4ad73e478a80107a13de2bf0df0a376" Jul 15 19:59:35.842949 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.842914 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m"] Jul 15 19:59:35.843436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.843208 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="492305f0-45da-48f3-b3bb-9451e651cd67" containerName="main" Jul 15 19:59:35.843436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.843218 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="492305f0-45da-48f3-b3bb-9451e651cd67" containerName="main" Jul 15 19:59:35.843436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.843232 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" containerName="main" Jul 15 19:59:35.843436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.843238 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" containerName="main" Jul 15 19:59:35.843436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.843283 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="eff8a0ac-4c5b-444b-a869-8ed29eaa9b46" containerName="main" Jul 15 19:59:35.843436 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.843290 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="492305f0-45da-48f3-b3bb-9451e651cd67" containerName="main" Jul 15 19:59:35.846175 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.846160 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:35.851336 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.851313 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"kube-root-ca.crt\"" Jul 15 19:59:35.851858 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.851840 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"default-dockercfg-djcbj\"" Jul 15 19:59:35.852605 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.852307 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"sub-test-model-fcd095-kserve-self-signed-certs\"" Jul 15 19:59:35.856422 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.856404 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"openshift-service-ca.crt\"" Jul 15 19:59:35.858566 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.858516 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 19:59:35.868784 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.868762 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m"] Jul 15 19:59:35.964048 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.964004 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-home\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:35.964237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.964064 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p479b\" (UniqueName: \"kubernetes.io/projected/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-kube-api-access-p479b\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:35.964237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.964108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-dshm\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:35.964237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.964131 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tmp-dir\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:35.964237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.964159 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-model-cache\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:35.964237 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:35.964197 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tls-certs\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.064790 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.064734 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-home\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.064989 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.064807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p479b\" (UniqueName: \"kubernetes.io/projected/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-kube-api-access-p479b\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.064989 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.064836 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-dshm\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.064989 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.064856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tmp-dir\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.065120 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.065002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-model-cache\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.065120 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.065064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tls-certs\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.065217 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.065145 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-home\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.065217 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.065190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tmp-dir\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.065381 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.065335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-model-cache\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.067090 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.067064 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-dshm\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.067518 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.067497 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tls-certs\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.079707 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.079680 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p479b\" (UniqueName: \"kubernetes.io/projected/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-kube-api-access-p479b\") pod \"sub-test-model-fcd095-kserve-5477cc8d49-2kb7m\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.155207 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.155119 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:36.296028 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.295994 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m"] Jul 15 19:59:36.298980 ip-10-0-132-25 kubenswrapper[2568]: W0715 19:59:36.298949 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ba0ab9e_be23_4e32_8ad7_46f2398389c2.slice/crio-975d446166a829b94971451f28c1b8a6fea361600950a5ecf354d7d9578fec7f WatchSource:0}: Error finding container 975d446166a829b94971451f28c1b8a6fea361600950a5ecf354d7d9578fec7f: Status 404 returned error can't find the container with id 975d446166a829b94971451f28c1b8a6fea361600950a5ecf354d7d9578fec7f Jul 15 19:59:36.931504 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.931464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" event={"ID":"3ba0ab9e-be23-4e32-8ad7-46f2398389c2","Type":"ContainerStarted","Data":"d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1"} Jul 15 19:59:36.931901 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:36.931515 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" event={"ID":"3ba0ab9e-be23-4e32-8ad7-46f2398389c2","Type":"ContainerStarted","Data":"975d446166a829b94971451f28c1b8a6fea361600950a5ecf354d7d9578fec7f"} Jul 15 19:59:40.347412 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:40.347336 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" podStartSLOduration=5.347319593 podStartE2EDuration="5.347319593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 19:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 19:59:36.96713149 +0000 UTC m=+2711.158009294" watchObservedRunningTime="2026-07-15 19:59:40.347319593 +0000 UTC m=+2714.538197398" Jul 15 19:59:40.348754 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:40.348734 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 19:59:46.155471 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:46.155417 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:46.155957 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:46.155496 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:46.167726 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:46.167702 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:46.990285 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:46.990257 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 19:59:59.714699 ip-10-0-132-25 kubenswrapper[2568]: I0715 19:59:59.714657 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:00:15.220445 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:15.220413 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:00:22.047522 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:22.047479 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49"] Jul 15 20:00:22.071506 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:22.071482 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:23.182881 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:23.182845 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m"] Jul 15 20:00:23.213964 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:23.213912 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:33.836717 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:33.836680 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:00:37.066713 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.066650 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" podUID="867a345b-fa3a-487b-8c36-bc91bd099495" containerName="main" containerID="cri-o://067b2af74c89f5ce104456af2e96c7e0c6f797db79d35fffbba5e86c71378f73" gracePeriod=45 Jul 15 20:00:37.221058 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.221028 2568 generic.go:356] "Generic (PLEG): container finished" podID="867a345b-fa3a-487b-8c36-bc91bd099495" containerID="067b2af74c89f5ce104456af2e96c7e0c6f797db79d35fffbba5e86c71378f73" exitCode=0 Jul 15 20:00:37.221231 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.221103 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" event={"ID":"867a345b-fa3a-487b-8c36-bc91bd099495","Type":"ContainerDied","Data":"067b2af74c89f5ce104456af2e96c7e0c6f797db79d35fffbba5e86c71378f73"} Jul 15 20:00:37.317814 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.317760 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 20:00:37.481730 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.481698 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867a345b-fa3a-487b-8c36-bc91bd099495-tls-certs\") pod \"867a345b-fa3a-487b-8c36-bc91bd099495\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " Jul 15 20:00:37.481935 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.481758 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-dshm\") pod \"867a345b-fa3a-487b-8c36-bc91bd099495\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " Jul 15 20:00:37.481935 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.481776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-model-cache\") pod \"867a345b-fa3a-487b-8c36-bc91bd099495\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " Jul 15 20:00:37.481935 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.481819 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-tmp-dir\") pod \"867a345b-fa3a-487b-8c36-bc91bd099495\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " Jul 15 20:00:37.481935 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.481864 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-home\") pod \"867a345b-fa3a-487b-8c36-bc91bd099495\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " Jul 15 20:00:37.481935 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.481903 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9djng\" (UniqueName: \"kubernetes.io/projected/867a345b-fa3a-487b-8c36-bc91bd099495-kube-api-access-9djng\") pod \"867a345b-fa3a-487b-8c36-bc91bd099495\" (UID: \"867a345b-fa3a-487b-8c36-bc91bd099495\") " Jul 15 20:00:37.482193 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.482100 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "867a345b-fa3a-487b-8c36-bc91bd099495" (UID: "867a345b-fa3a-487b-8c36-bc91bd099495"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:37.482193 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.482111 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-model-cache" (OuterVolumeSpecName: "model-cache") pod "867a345b-fa3a-487b-8c36-bc91bd099495" (UID: "867a345b-fa3a-487b-8c36-bc91bd099495"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:37.482262 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.482179 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-home" (OuterVolumeSpecName: "home") pod "867a345b-fa3a-487b-8c36-bc91bd099495" (UID: "867a345b-fa3a-487b-8c36-bc91bd099495"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:37.484090 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.484067 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867a345b-fa3a-487b-8c36-bc91bd099495-kube-api-access-9djng" (OuterVolumeSpecName: "kube-api-access-9djng") pod "867a345b-fa3a-487b-8c36-bc91bd099495" (UID: "867a345b-fa3a-487b-8c36-bc91bd099495"). InnerVolumeSpecName "kube-api-access-9djng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:00:37.484368 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.484328 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-dshm" (OuterVolumeSpecName: "dshm") pod "867a345b-fa3a-487b-8c36-bc91bd099495" (UID: "867a345b-fa3a-487b-8c36-bc91bd099495"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:37.484437 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.484362 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867a345b-fa3a-487b-8c36-bc91bd099495-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "867a345b-fa3a-487b-8c36-bc91bd099495" (UID: "867a345b-fa3a-487b-8c36-bc91bd099495"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:00:37.583094 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.583006 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:37.583094 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.583045 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9djng\" (UniqueName: \"kubernetes.io/projected/867a345b-fa3a-487b-8c36-bc91bd099495-kube-api-access-9djng\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:37.583094 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.583060 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/867a345b-fa3a-487b-8c36-bc91bd099495-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:37.583094 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.583073 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:37.583094 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.583084 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:37.583094 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:37.583095 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/867a345b-fa3a-487b-8c36-bc91bd099495-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:38.202298 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.202262 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" podUID="3ba0ab9e-be23-4e32-8ad7-46f2398389c2" containerName="main" containerID="cri-o://d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1" gracePeriod=45 Jul 15 20:00:38.227424 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.227336 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" Jul 15 20:00:38.228452 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.228430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49" event={"ID":"867a345b-fa3a-487b-8c36-bc91bd099495","Type":"ContainerDied","Data":"74617eb8dd79e0f4c100440b03771e4f74457764bf3a42c1c940a1493bb7a412"} Jul 15 20:00:38.228537 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.228467 2568 scope.go:117] "RemoveContainer" containerID="067b2af74c89f5ce104456af2e96c7e0c6f797db79d35fffbba5e86c71378f73" Jul 15 20:00:38.283617 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.283591 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49"] Jul 15 20:00:38.296455 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.296430 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/sub-test-model-77bf75-kserve-585868c995-5lz49"] Jul 15 20:00:38.447326 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.447296 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 20:00:38.590960 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.590927 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-model-cache\") pod \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " Jul 15 20:00:38.590960 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.590964 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-dshm\") pod \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " Jul 15 20:00:38.591230 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591019 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tmp-dir\") pod \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " Jul 15 20:00:38.591230 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591035 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tls-certs\") pod \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " Jul 15 20:00:38.591230 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591083 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p479b\" (UniqueName: \"kubernetes.io/projected/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-kube-api-access-p479b\") pod \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " Jul 15 20:00:38.591230 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591166 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-home\") pod \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\" (UID: \"3ba0ab9e-be23-4e32-8ad7-46f2398389c2\") " Jul 15 20:00:38.591230 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591197 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ba0ab9e-be23-4e32-8ad7-46f2398389c2" (UID: "3ba0ab9e-be23-4e32-8ad7-46f2398389c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:38.591508 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591303 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ba0ab9e-be23-4e32-8ad7-46f2398389c2" (UID: "3ba0ab9e-be23-4e32-8ad7-46f2398389c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:38.591508 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591435 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:38.591508 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591455 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:38.591508 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.591450 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-home" (OuterVolumeSpecName: "home") pod "3ba0ab9e-be23-4e32-8ad7-46f2398389c2" (UID: "3ba0ab9e-be23-4e32-8ad7-46f2398389c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:38.593289 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.593266 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ba0ab9e-be23-4e32-8ad7-46f2398389c2" (UID: "3ba0ab9e-be23-4e32-8ad7-46f2398389c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:00:38.593381 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.593312 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-kube-api-access-p479b" (OuterVolumeSpecName: "kube-api-access-p479b") pod "3ba0ab9e-be23-4e32-8ad7-46f2398389c2" (UID: "3ba0ab9e-be23-4e32-8ad7-46f2398389c2"). InnerVolumeSpecName "kube-api-access-p479b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:00:38.593381 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.593372 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-dshm" (OuterVolumeSpecName: "dshm") pod "3ba0ab9e-be23-4e32-8ad7-46f2398389c2" (UID: "3ba0ab9e-be23-4e32-8ad7-46f2398389c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:00:38.692623 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.692589 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:38.692623 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.692618 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p479b\" (UniqueName: \"kubernetes.io/projected/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-kube-api-access-p479b\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:38.692623 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.692628 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:38.692859 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:38.692637 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ba0ab9e-be23-4e32-8ad7-46f2398389c2-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:00:39.233287 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.233253 2568 generic.go:356] "Generic (PLEG): container finished" podID="3ba0ab9e-be23-4e32-8ad7-46f2398389c2" containerID="d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1" exitCode=0 Jul 15 20:00:39.233784 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.233295 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" Jul 15 20:00:39.233784 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.233366 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" event={"ID":"3ba0ab9e-be23-4e32-8ad7-46f2398389c2","Type":"ContainerDied","Data":"d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1"} Jul 15 20:00:39.233784 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.233402 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m" event={"ID":"3ba0ab9e-be23-4e32-8ad7-46f2398389c2","Type":"ContainerDied","Data":"975d446166a829b94971451f28c1b8a6fea361600950a5ecf354d7d9578fec7f"} Jul 15 20:00:39.233784 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.233418 2568 scope.go:117] "RemoveContainer" containerID="d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1" Jul 15 20:00:39.242323 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.242307 2568 scope.go:117] "RemoveContainer" containerID="d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1" Jul 15 20:00:39.242595 ip-10-0-132-25 kubenswrapper[2568]: E0715 20:00:39.242574 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1\": container with ID starting with d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1 not found: ID does not exist" containerID="d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1" Jul 15 20:00:39.242644 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.242603 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1"} err="failed to get container status \"d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1\": rpc error: code = NotFound desc = could not find container \"d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1\": container with ID starting with d7607c9a57262d0b65f0512cd9acbe44a00f47889c9e6a9e09009a0f6d59d3f1 not found: ID does not exist" Jul 15 20:00:39.265469 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.265447 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m"] Jul 15 20:00:39.273909 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:39.273890 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/sub-test-model-fcd095-kserve-5477cc8d49-2kb7m"] Jul 15 20:00:40.228642 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:40.228612 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba0ab9e-be23-4e32-8ad7-46f2398389c2" path="/var/lib/kubelet/pods/3ba0ab9e-be23-4e32-8ad7-46f2398389c2/volumes" Jul 15 20:00:40.229005 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:40.228991 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="867a345b-fa3a-487b-8c36-bc91bd099495" path="/var/lib/kubelet/pods/867a345b-fa3a-487b-8c36-bc91bd099495/volumes" Jul 15 20:00:44.501151 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.501122 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr"] Jul 15 20:00:44.501553 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.501457 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867a345b-fa3a-487b-8c36-bc91bd099495" containerName="main" Jul 15 20:00:44.501553 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.501469 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="867a345b-fa3a-487b-8c36-bc91bd099495" containerName="main" Jul 15 20:00:44.501553 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.501479 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba0ab9e-be23-4e32-8ad7-46f2398389c2" containerName="main" Jul 15 20:00:44.501553 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.501484 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba0ab9e-be23-4e32-8ad7-46f2398389c2" containerName="main" Jul 15 20:00:44.501553 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.501547 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba0ab9e-be23-4e32-8ad7-46f2398389c2" containerName="main" Jul 15 20:00:44.501553 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.501557 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="867a345b-fa3a-487b-8c36-bc91bd099495" containerName="main" Jul 15 20:00:44.505770 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.505753 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.508489 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.508472 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"default-dockercfg-7kdrr\"" Jul 15 20:00:44.509476 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.509454 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"openshift-service-ca.crt\"" Jul 15 20:00:44.509569 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.509483 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"rate-test-model-77bf75-kserve-self-signed-certs\"" Jul 15 20:00:44.509868 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.509853 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-77bf75\"/\"kube-root-ca.crt\"" Jul 15 20:00:44.525921 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.525895 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr"] Jul 15 20:00:44.594922 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.594886 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:44.643004 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.642978 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-model-cache\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.643171 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.643014 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-home\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.643171 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.643037 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbdt\" (UniqueName: \"kubernetes.io/projected/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-kube-api-access-8tbdt\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.643171 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.643117 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tmp-dir\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.643171 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.643153 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tls-certs\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.643362 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.643201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-dshm\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744230 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744195 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-model-cache\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744230 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-home\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744490 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbdt\" (UniqueName: \"kubernetes.io/projected/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-kube-api-access-8tbdt\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744490 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tmp-dir\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744490 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tls-certs\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744490 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744422 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-dshm\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744725 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744666 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-home\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744725 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744691 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tmp-dir\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.744843 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.744734 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-model-cache\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.746631 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.746610 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-dshm\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.746794 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.746775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tls-certs\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.762310 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.762228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbdt\" (UniqueName: \"kubernetes.io/projected/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-kube-api-access-8tbdt\") pod \"rate-test-model-77bf75-kserve-b7c7f5f47-728kr\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.814227 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.814193 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:44.951194 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:00:44.951163 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e9cca96_7893_4edd_9f14_e4cdd27c03ca.slice/crio-1e3fcd60530ebc5b9fce16f9a3ee21a90fadc6cfbfae254e73d003cb3700e37c WatchSource:0}: Error finding container 1e3fcd60530ebc5b9fce16f9a3ee21a90fadc6cfbfae254e73d003cb3700e37c: Status 404 returned error can't find the container with id 1e3fcd60530ebc5b9fce16f9a3ee21a90fadc6cfbfae254e73d003cb3700e37c Jul 15 20:00:44.952515 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:44.952489 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr"] Jul 15 20:00:45.263924 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:45.263891 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" event={"ID":"0e9cca96-7893-4edd-9f14-e4cdd27c03ca","Type":"ContainerStarted","Data":"c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa"} Jul 15 20:00:45.263924 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:45.263927 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" event={"ID":"0e9cca96-7893-4edd-9f14-e4cdd27c03ca","Type":"ContainerStarted","Data":"1e3fcd60530ebc5b9fce16f9a3ee21a90fadc6cfbfae254e73d003cb3700e37c"} Jul 15 20:00:45.289804 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:45.289758 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" podStartSLOduration=1.289740292 podStartE2EDuration="1.289740292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:00:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:00:45.289240088 +0000 UTC m=+2779.480117890" watchObservedRunningTime="2026-07-15 20:00:45.289740292 +0000 UTC m=+2779.480618099" Jul 15 20:00:50.564935 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:50.564897 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:00:52.747464 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:52.747402 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:52.833456 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:52.833423 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:00:54.815051 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:54.815011 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:54.815483 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:54.815076 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:54.827738 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:54.827712 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:00:55.322138 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:00:55.322111 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:01:02.412565 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:02.412533 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:04.038198 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.038159 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks"] Jul 15 20:01:04.041460 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.041445 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.045084 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.045064 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"openshift-service-ca.crt\"" Jul 15 20:01:04.045939 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.045925 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"kube-root-ca.crt\"" Jul 15 20:01:04.046896 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.046880 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"default-dockercfg-djcbj\"" Jul 15 20:01:04.047223 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.047206 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fcd095\"/\"rate-test-model-fcd095-kserve-self-signed-certs\"" Jul 15 20:01:04.072501 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.072480 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks"] Jul 15 20:01:04.095981 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.095962 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-model-cache\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.096071 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.095994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18af6b0-b524-49be-86b3-7fd8e2787a41-tls-certs\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.096071 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.096029 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd72s\" (UniqueName: \"kubernetes.io/projected/f18af6b0-b524-49be-86b3-7fd8e2787a41-kube-api-access-pd72s\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.096147 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.096087 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-dshm\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.096182 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.096143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-tmp-dir\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.096182 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.096163 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-home\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.196915 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.196886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-model-cache\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197045 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.196922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18af6b0-b524-49be-86b3-7fd8e2787a41-tls-certs\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197045 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.196948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd72s\" (UniqueName: \"kubernetes.io/projected/f18af6b0-b524-49be-86b3-7fd8e2787a41-kube-api-access-pd72s\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197045 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.196988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-dshm\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197045 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.197020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-tmp-dir\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197045 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.197036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-home\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197304 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.197283 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-model-cache\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197401 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.197365 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-home\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.197467 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.197446 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-tmp-dir\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.199281 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.199254 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-dshm\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.199421 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.199377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18af6b0-b524-49be-86b3-7fd8e2787a41-tls-certs\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.210971 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.210948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd72s\" (UniqueName: \"kubernetes.io/projected/f18af6b0-b524-49be-86b3-7fd8e2787a41-kube-api-access-pd72s\") pod \"rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.281067 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.281046 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:04.351268 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.351200 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:04.448375 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.448311 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:04.537272 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:04.537246 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks"] Jul 15 20:01:04.538823 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:01:04.538802 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf18af6b0_b524_49be_86b3_7fd8e2787a41.slice/crio-35f5309152eefa76da84cb1306e908570a7862d25b029737cf42ee5676fc70cc WatchSource:0}: Error finding container 35f5309152eefa76da84cb1306e908570a7862d25b029737cf42ee5676fc70cc: Status 404 returned error can't find the container with id 35f5309152eefa76da84cb1306e908570a7862d25b029737cf42ee5676fc70cc Jul 15 20:01:05.361228 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:05.361189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" event={"ID":"f18af6b0-b524-49be-86b3-7fd8e2787a41","Type":"ContainerStarted","Data":"d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41"} Jul 15 20:01:05.361663 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:05.361235 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" event={"ID":"f18af6b0-b524-49be-86b3-7fd8e2787a41","Type":"ContainerStarted","Data":"35f5309152eefa76da84cb1306e908570a7862d25b029737cf42ee5676fc70cc"} Jul 15 20:01:05.399799 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:05.399748 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" podStartSLOduration=1.3997308419999999 podStartE2EDuration="1.399730842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:01:05.398746314 +0000 UTC m=+2799.589624118" watchObservedRunningTime="2026-07-15 20:01:05.399730842 +0000 UTC m=+2799.590608647" Jul 15 20:01:09.081506 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:09.081470 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:01:14.352163 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:14.352132 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:14.352574 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:14.352179 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:14.364746 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:14.364718 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:14.413400 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:14.413378 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:01:23.924118 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:23.924082 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:01:32.231177 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:32.231145 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:32.299227 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:32.299191 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:41.438191 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:41.438149 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:01:47.251875 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:47.251836 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr"] Jul 15 20:01:47.281709 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:47.281672 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:48.227982 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:48.227954 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks"] Jul 15 20:01:48.244401 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:48.244364 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:01:49.202974 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.202941 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140644952668848"] Jul 15 20:01:49.206221 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.206204 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:49.211728 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.211706 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hwmqg\"" Jul 15 20:01:49.219406 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.219384 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952668848"] Jul 15 20:01:49.366264 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.366224 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqhwh\" (UniqueName: \"kubernetes.io/projected/8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b-kube-api-access-tqhwh\") pod \"test-curl-7442-140644952668848\" (UID: \"8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b\") " pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:49.467268 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.467170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqhwh\" (UniqueName: \"kubernetes.io/projected/8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b-kube-api-access-tqhwh\") pod \"test-curl-7442-140644952668848\" (UID: \"8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b\") " pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:49.479462 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.479431 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqhwh\" (UniqueName: \"kubernetes.io/projected/8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b-kube-api-access-tqhwh\") pod \"test-curl-7442-140644952668848\" (UID: \"8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b\") " pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:49.515033 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.515009 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:49.855114 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.855089 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952668848"] Jul 15 20:01:49.856436 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:01:49.856402 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8acdb4cd_0eb3_4aa6_8efb_0a4b3a09961b.slice/crio-3aeed3638fa6c1a982bb9bd121af3e96ba5250fbc9e3fe1009dec23869328fbf WatchSource:0}: Error finding container 3aeed3638fa6c1a982bb9bd121af3e96ba5250fbc9e3fe1009dec23869328fbf: Status 404 returned error can't find the container with id 3aeed3638fa6c1a982bb9bd121af3e96ba5250fbc9e3fe1009dec23869328fbf Jul 15 20:01:49.857738 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:49.857720 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 20:01:50.581143 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:50.581112 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952668848" event={"ID":"8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b","Type":"ContainerStarted","Data":"3aeed3638fa6c1a982bb9bd121af3e96ba5250fbc9e3fe1009dec23869328fbf"} Jul 15 20:01:51.587400 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:51.587370 2568 generic.go:356] "Generic (PLEG): container finished" podID="8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b" containerID="5835e7b27d4bf63df31c9949500247c6c68258bfbf127391af55f43d7623206c" exitCode=0 Jul 15 20:01:51.587787 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:51.587419 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952668848" event={"ID":"8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b","Type":"ContainerDied","Data":"5835e7b27d4bf63df31c9949500247c6c68258bfbf127391af55f43d7623206c"} Jul 15 20:01:52.724375 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.724336 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:52.767976 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.767944 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140644952668848_8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b/test-curl-7442-140644952668848/0.log" Jul 15 20:01:52.805186 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.805147 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140644952668848"] Jul 15 20:01:52.812359 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.812323 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140644952668848"] Jul 15 20:01:52.897984 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.897894 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqhwh\" (UniqueName: \"kubernetes.io/projected/8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b-kube-api-access-tqhwh\") pod \"8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b\" (UID: \"8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b\") " Jul 15 20:01:52.900035 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.900003 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b-kube-api-access-tqhwh" (OuterVolumeSpecName: "kube-api-access-tqhwh") pod "8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b" (UID: "8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b"). InnerVolumeSpecName "kube-api-access-tqhwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:01:52.941284 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.941254 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140644952669104"] Jul 15 20:01:52.941608 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.941594 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b" containerName="test-curl-7442-140644952668848" Jul 15 20:01:52.941668 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.941611 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b" containerName="test-curl-7442-140644952668848" Jul 15 20:01:52.941706 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.941695 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b" containerName="test-curl-7442-140644952668848" Jul 15 20:01:52.944557 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.944543 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952669104" Jul 15 20:01:52.961027 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.961002 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952669104"] Jul 15 20:01:52.998913 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:52.998885 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqhwh\" (UniqueName: \"kubernetes.io/projected/8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b-kube-api-access-tqhwh\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:01:53.099529 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.099494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnzb\" (UniqueName: \"kubernetes.io/projected/d5918a93-b4c4-4e21-bd61-117404bc8cd0-kube-api-access-hxnzb\") pod \"test-curl-7442-140644952669104\" (UID: \"d5918a93-b4c4-4e21-bd61-117404bc8cd0\") " pod="opendatahub/test-curl-7442-140644952669104" Jul 15 20:01:53.200137 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.200035 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnzb\" (UniqueName: \"kubernetes.io/projected/d5918a93-b4c4-4e21-bd61-117404bc8cd0-kube-api-access-hxnzb\") pod \"test-curl-7442-140644952669104\" (UID: \"d5918a93-b4c4-4e21-bd61-117404bc8cd0\") " pod="opendatahub/test-curl-7442-140644952669104" Jul 15 20:01:53.211676 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.211649 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnzb\" (UniqueName: \"kubernetes.io/projected/d5918a93-b4c4-4e21-bd61-117404bc8cd0-kube-api-access-hxnzb\") pod \"test-curl-7442-140644952669104\" (UID: \"d5918a93-b4c4-4e21-bd61-117404bc8cd0\") " pod="opendatahub/test-curl-7442-140644952669104" Jul 15 20:01:53.253008 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.252972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952669104" Jul 15 20:01:53.388408 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.388378 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952669104"] Jul 15 20:01:53.389403 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:01:53.389375 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5918a93_b4c4_4e21_bd61_117404bc8cd0.slice/crio-25647442190c86d6a49c5aeda2f83da19273b305cf57aaeaa1e3b05fab2a499b WatchSource:0}: Error finding container 25647442190c86d6a49c5aeda2f83da19273b305cf57aaeaa1e3b05fab2a499b: Status 404 returned error can't find the container with id 25647442190c86d6a49c5aeda2f83da19273b305cf57aaeaa1e3b05fab2a499b Jul 15 20:01:53.597613 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.597583 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:53.597774 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.597586 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aeed3638fa6c1a982bb9bd121af3e96ba5250fbc9e3fe1009dec23869328fbf" Jul 15 20:01:53.598829 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:53.598805 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952669104" event={"ID":"d5918a93-b4c4-4e21-bd61-117404bc8cd0","Type":"ContainerStarted","Data":"25647442190c86d6a49c5aeda2f83da19273b305cf57aaeaa1e3b05fab2a499b"} Jul 15 20:01:53.608148 ip-10-0-132-25 kubenswrapper[2568]: E0715 20:01:53.608124 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7442-140644952668848\" is forbidden: User \"system:node:ip-10-0-132-25.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-25.ec2.internal' and this object" podUID="8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b" pod="opendatahub/test-curl-7442-140644952668848" Jul 15 20:01:54.228532 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:54.228502 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b" path="/var/lib/kubelet/pods/8acdb4cd-0eb3-4aa6-8efb-0a4b3a09961b/volumes" Jul 15 20:01:54.605428 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:54.605393 2568 generic.go:356] "Generic (PLEG): container finished" podID="d5918a93-b4c4-4e21-bd61-117404bc8cd0" containerID="71ca985bfe5c98803e6d5f729947cdc23f8bfe40aeb76bd690d3e5fbf3e37bb7" exitCode=0 Jul 15 20:01:54.605615 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:54.605478 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952669104" event={"ID":"d5918a93-b4c4-4e21-bd61-117404bc8cd0","Type":"ContainerDied","Data":"71ca985bfe5c98803e6d5f729947cdc23f8bfe40aeb76bd690d3e5fbf3e37bb7"} Jul 15 20:01:55.737918 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.737888 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952669104" Jul 15 20:01:55.768522 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.768494 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140644952669104_d5918a93-b4c4-4e21-bd61-117404bc8cd0/test-curl-7442-140644952669104/0.log" Jul 15 20:01:55.802849 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.802776 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140644952669104"] Jul 15 20:01:55.804905 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.804881 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140644952669104"] Jul 15 20:01:55.925058 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.924964 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxnzb\" (UniqueName: \"kubernetes.io/projected/d5918a93-b4c4-4e21-bd61-117404bc8cd0-kube-api-access-hxnzb\") pod \"d5918a93-b4c4-4e21-bd61-117404bc8cd0\" (UID: \"d5918a93-b4c4-4e21-bd61-117404bc8cd0\") " Jul 15 20:01:55.927173 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.927147 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5918a93-b4c4-4e21-bd61-117404bc8cd0-kube-api-access-hxnzb" (OuterVolumeSpecName: "kube-api-access-hxnzb") pod "d5918a93-b4c4-4e21-bd61-117404bc8cd0" (UID: "d5918a93-b4c4-4e21-bd61-117404bc8cd0"). InnerVolumeSpecName "kube-api-access-hxnzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:01:55.979810 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.979771 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:55.980169 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.980135 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5918a93-b4c4-4e21-bd61-117404bc8cd0" containerName="test-curl-7442-140644952669104" Jul 15 20:01:55.980254 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.980171 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5918a93-b4c4-4e21-bd61-117404bc8cd0" containerName="test-curl-7442-140644952669104" Jul 15 20:01:55.980306 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.980276 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5918a93-b4c4-4e21-bd61-117404bc8cd0" containerName="test-curl-7442-140644952669104" Jul 15 20:01:55.983271 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:55.983250 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:56.003013 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.002986 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:56.026570 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.026542 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnzb\" (UniqueName: \"kubernetes.io/projected/d5918a93-b4c4-4e21-bd61-117404bc8cd0-kube-api-access-hxnzb\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:01:56.127738 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.127703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjjs\" (UniqueName: \"kubernetes.io/projected/5f2b1c9d-402d-43db-8932-2db56c352724-kube-api-access-pwjjs\") pod \"test-curl-7442-140644952707120\" (UID: \"5f2b1c9d-402d-43db-8932-2db56c352724\") " pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:56.228579 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.228506 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5918a93-b4c4-4e21-bd61-117404bc8cd0" path="/var/lib/kubelet/pods/d5918a93-b4c4-4e21-bd61-117404bc8cd0/volumes" Jul 15 20:01:56.228982 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.228964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjjs\" (UniqueName: \"kubernetes.io/projected/5f2b1c9d-402d-43db-8932-2db56c352724-kube-api-access-pwjjs\") pod \"test-curl-7442-140644952707120\" (UID: \"5f2b1c9d-402d-43db-8932-2db56c352724\") " pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:56.241148 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.241127 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjjs\" (UniqueName: \"kubernetes.io/projected/5f2b1c9d-402d-43db-8932-2db56c352724-kube-api-access-pwjjs\") pod \"test-curl-7442-140644952707120\" (UID: \"5f2b1c9d-402d-43db-8932-2db56c352724\") " pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:56.292094 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.292059 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:56.433301 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.433274 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:56.434997 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:01:56.434971 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f2b1c9d_402d_43db_8932_2db56c352724.slice/crio-3b1daa7e49292b5024b6898b21d7ee5d880ea3eddf8892083167e5ba7476d1ad WatchSource:0}: Error finding container 3b1daa7e49292b5024b6898b21d7ee5d880ea3eddf8892083167e5ba7476d1ad: Status 404 returned error can't find the container with id 3b1daa7e49292b5024b6898b21d7ee5d880ea3eddf8892083167e5ba7476d1ad Jul 15 20:01:56.616599 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.616567 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952669104" Jul 15 20:01:56.616805 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.616599 2568 scope.go:117] "RemoveContainer" containerID="71ca985bfe5c98803e6d5f729947cdc23f8bfe40aeb76bd690d3e5fbf3e37bb7" Jul 15 20:01:56.617908 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:56.617776 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952707120" event={"ID":"5f2b1c9d-402d-43db-8932-2db56c352724","Type":"ContainerStarted","Data":"3b1daa7e49292b5024b6898b21d7ee5d880ea3eddf8892083167e5ba7476d1ad"} Jul 15 20:01:57.625268 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:57.625229 2568 generic.go:356] "Generic (PLEG): container finished" podID="5f2b1c9d-402d-43db-8932-2db56c352724" containerID="de5df57a6b928600fb1045ed48f9c9c6beba1d6720e4be4afcac7f90b2836ac6" exitCode=0 Jul 15 20:01:57.625701 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:57.625296 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952707120" event={"ID":"5f2b1c9d-402d-43db-8932-2db56c352724","Type":"ContainerDied","Data":"de5df57a6b928600fb1045ed48f9c9c6beba1d6720e4be4afcac7f90b2836ac6"} Jul 15 20:01:58.763279 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.763248 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:58.794989 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.794962 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140644952707120_5f2b1c9d-402d-43db-8932-2db56c352724/test-curl-7442-140644952707120/0.log" Jul 15 20:01:58.825097 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.825062 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:58.834326 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.834298 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:58.952222 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.952126 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwjjs\" (UniqueName: \"kubernetes.io/projected/5f2b1c9d-402d-43db-8932-2db56c352724-kube-api-access-pwjjs\") pod \"5f2b1c9d-402d-43db-8932-2db56c352724\" (UID: \"5f2b1c9d-402d-43db-8932-2db56c352724\") " Jul 15 20:01:58.954232 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.954206 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2b1c9d-402d-43db-8932-2db56c352724-kube-api-access-pwjjs" (OuterVolumeSpecName: "kube-api-access-pwjjs") pod "5f2b1c9d-402d-43db-8932-2db56c352724" (UID: "5f2b1c9d-402d-43db-8932-2db56c352724"). InnerVolumeSpecName "kube-api-access-pwjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:01:58.965685 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.965661 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:58.965997 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.965980 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f2b1c9d-402d-43db-8932-2db56c352724" containerName="test-curl-7442-140644952707120" Jul 15 20:01:58.966074 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.966000 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f2b1c9d-402d-43db-8932-2db56c352724" containerName="test-curl-7442-140644952707120" Jul 15 20:01:58.966133 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.966112 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f2b1c9d-402d-43db-8932-2db56c352724" containerName="test-curl-7442-140644952707120" Jul 15 20:01:58.968936 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.968918 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:58.978595 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:58.978576 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:59.053603 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.053561 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjjs\" (UniqueName: \"kubernetes.io/projected/5f2b1c9d-402d-43db-8932-2db56c352724-kube-api-access-pwjjs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:01:59.154263 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.154229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvsr\" (UniqueName: \"kubernetes.io/projected/4f074c07-33f7-4872-8262-9e0182516d43-kube-api-access-brvsr\") pod \"test-curl-7442-140644952707120\" (UID: \"4f074c07-33f7-4872-8262-9e0182516d43\") " pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:59.255028 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.254997 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvsr\" (UniqueName: \"kubernetes.io/projected/4f074c07-33f7-4872-8262-9e0182516d43-kube-api-access-brvsr\") pod \"test-curl-7442-140644952707120\" (UID: \"4f074c07-33f7-4872-8262-9e0182516d43\") " pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:59.265225 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.265190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvsr\" (UniqueName: \"kubernetes.io/projected/4f074c07-33f7-4872-8262-9e0182516d43-kube-api-access-brvsr\") pod \"test-curl-7442-140644952707120\" (UID: \"4f074c07-33f7-4872-8262-9e0182516d43\") " pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:59.278084 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.278058 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:59.313069 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.313041 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:01:59.407399 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.407376 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:01:59.408993 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:01:59.408969 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f074c07_33f7_4872_8262_9e0182516d43.slice/crio-bd576979004db73def1133623f9890b54b6642a912eb4da64a4e12b3a8295f4d WatchSource:0}: Error finding container bd576979004db73def1133623f9890b54b6642a912eb4da64a4e12b3a8295f4d: Status 404 returned error can't find the container with id bd576979004db73def1133623f9890b54b6642a912eb4da64a4e12b3a8295f4d Jul 15 20:01:59.635714 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.635628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952707120" event={"ID":"4f074c07-33f7-4872-8262-9e0182516d43","Type":"ContainerStarted","Data":"bd576979004db73def1133623f9890b54b6642a912eb4da64a4e12b3a8295f4d"} Jul 15 20:01:59.636793 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.636775 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b1daa7e49292b5024b6898b21d7ee5d880ea3eddf8892083167e5ba7476d1ad" Jul 15 20:01:59.636793 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.636787 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:01:59.652059 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:01:59.652028 2568 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7442-140644952707120" oldPodUID="5f2b1c9d-402d-43db-8932-2db56c352724" podUID="4f074c07-33f7-4872-8262-9e0182516d43" Jul 15 20:02:00.229991 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:00.229948 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f2b1c9d-402d-43db-8932-2db56c352724" path="/var/lib/kubelet/pods/5f2b1c9d-402d-43db-8932-2db56c352724/volumes" Jul 15 20:02:00.643244 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:00.643210 2568 generic.go:356] "Generic (PLEG): container finished" podID="4f074c07-33f7-4872-8262-9e0182516d43" containerID="d2884643d1732beda9cbb81cbd48bbf52e7e7b980aeb6dcdecb513a931c80458" exitCode=0 Jul 15 20:02:00.643435 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:00.643282 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952707120" event={"ID":"4f074c07-33f7-4872-8262-9e0182516d43","Type":"ContainerDied","Data":"d2884643d1732beda9cbb81cbd48bbf52e7e7b980aeb6dcdecb513a931c80458"} Jul 15 20:02:01.776429 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.776401 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:02:01.806113 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.806070 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140644952707120_4f074c07-33f7-4872-8262-9e0182516d43/test-curl-7442-140644952707120/0.log" Jul 15 20:02:01.833101 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.833072 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:02:01.839386 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.839358 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140644952707120"] Jul 15 20:02:01.972051 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.971974 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:01.972283 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.972271 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f074c07-33f7-4872-8262-9e0182516d43" containerName="test-curl-7442-140644952707120" Jul 15 20:02:01.972326 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.972285 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f074c07-33f7-4872-8262-9e0182516d43" containerName="test-curl-7442-140644952707120" Jul 15 20:02:01.972375 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.972340 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f074c07-33f7-4872-8262-9e0182516d43" containerName="test-curl-7442-140644952707120" Jul 15 20:02:01.975215 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.975200 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:01.977519 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.977498 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brvsr\" (UniqueName: \"kubernetes.io/projected/4f074c07-33f7-4872-8262-9e0182516d43-kube-api-access-brvsr\") pod \"4f074c07-33f7-4872-8262-9e0182516d43\" (UID: \"4f074c07-33f7-4872-8262-9e0182516d43\") " Jul 15 20:02:01.977769 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.977750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nvwv\" (UniqueName: \"kubernetes.io/projected/17fbd7b3-e6e3-406a-9b25-d8ae8c581e60-kube-api-access-8nvwv\") pod \"test-curl-7442-140644952708128\" (UID: \"17fbd7b3-e6e3-406a-9b25-d8ae8c581e60\") " pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:01.979456 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.979436 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f074c07-33f7-4872-8262-9e0182516d43-kube-api-access-brvsr" (OuterVolumeSpecName: "kube-api-access-brvsr") pod "4f074c07-33f7-4872-8262-9e0182516d43" (UID: "4f074c07-33f7-4872-8262-9e0182516d43"). InnerVolumeSpecName "kube-api-access-brvsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:02:01.994872 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:01.994843 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:02.078579 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.078530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nvwv\" (UniqueName: \"kubernetes.io/projected/17fbd7b3-e6e3-406a-9b25-d8ae8c581e60-kube-api-access-8nvwv\") pod \"test-curl-7442-140644952708128\" (UID: \"17fbd7b3-e6e3-406a-9b25-d8ae8c581e60\") " pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:02.078774 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.078633 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvsr\" (UniqueName: \"kubernetes.io/projected/4f074c07-33f7-4872-8262-9e0182516d43-kube-api-access-brvsr\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:02.091872 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.091843 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nvwv\" (UniqueName: \"kubernetes.io/projected/17fbd7b3-e6e3-406a-9b25-d8ae8c581e60-kube-api-access-8nvwv\") pod \"test-curl-7442-140644952708128\" (UID: \"17fbd7b3-e6e3-406a-9b25-d8ae8c581e60\") " pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:02.229073 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.228992 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f074c07-33f7-4872-8262-9e0182516d43" path="/var/lib/kubelet/pods/4f074c07-33f7-4872-8262-9e0182516d43/volumes" Jul 15 20:02:02.271451 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.271410 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" podUID="0e9cca96-7893-4edd-9f14-e4cdd27c03ca" containerName="main" containerID="cri-o://c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa" gracePeriod=45 Jul 15 20:02:02.290024 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.289999 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:02.436103 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.436076 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:02.438691 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:02:02.438649 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17fbd7b3_e6e3_406a_9b25_d8ae8c581e60.slice/crio-66b5927fbbc4a4a0db884ee5f3a72ed250aba2ac9950f06ce2fbd10953f95100 WatchSource:0}: Error finding container 66b5927fbbc4a4a0db884ee5f3a72ed250aba2ac9950f06ce2fbd10953f95100: Status 404 returned error can't find the container with id 66b5927fbbc4a4a0db884ee5f3a72ed250aba2ac9950f06ce2fbd10953f95100 Jul 15 20:02:02.510810 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.510780 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:02:02.581956 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.581928 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tls-certs\") pod \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " Jul 15 20:02:02.582132 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.581969 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tbdt\" (UniqueName: \"kubernetes.io/projected/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-kube-api-access-8tbdt\") pod \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " Jul 15 20:02:02.582132 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.581991 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-dshm\") pod \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " Jul 15 20:02:02.582132 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.582024 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-home\") pod \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " Jul 15 20:02:02.582392 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.582371 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-home" (OuterVolumeSpecName: "home") pod "0e9cca96-7893-4edd-9f14-e4cdd27c03ca" (UID: "0e9cca96-7893-4edd-9f14-e4cdd27c03ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:02.584039 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.584013 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e9cca96-7893-4edd-9f14-e4cdd27c03ca" (UID: "0e9cca96-7893-4edd-9f14-e4cdd27c03ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:02:02.584151 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.584046 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-dshm" (OuterVolumeSpecName: "dshm") pod "0e9cca96-7893-4edd-9f14-e4cdd27c03ca" (UID: "0e9cca96-7893-4edd-9f14-e4cdd27c03ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:02.584151 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.584079 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-kube-api-access-8tbdt" (OuterVolumeSpecName: "kube-api-access-8tbdt") pod "0e9cca96-7893-4edd-9f14-e4cdd27c03ca" (UID: "0e9cca96-7893-4edd-9f14-e4cdd27c03ca"). InnerVolumeSpecName "kube-api-access-8tbdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:02:02.655491 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.655456 2568 generic.go:356] "Generic (PLEG): container finished" podID="0e9cca96-7893-4edd-9f14-e4cdd27c03ca" containerID="c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa" exitCode=0 Jul 15 20:02:02.655663 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.655507 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" Jul 15 20:02:02.655663 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.655553 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" event={"ID":"0e9cca96-7893-4edd-9f14-e4cdd27c03ca","Type":"ContainerDied","Data":"c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa"} Jul 15 20:02:02.655663 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.655596 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr" event={"ID":"0e9cca96-7893-4edd-9f14-e4cdd27c03ca","Type":"ContainerDied","Data":"1e3fcd60530ebc5b9fce16f9a3ee21a90fadc6cfbfae254e73d003cb3700e37c"} Jul 15 20:02:02.655663 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.655612 2568 scope.go:117] "RemoveContainer" containerID="c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa" Jul 15 20:02:02.656833 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.656766 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952707120" Jul 15 20:02:02.658027 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.658002 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952708128" event={"ID":"17fbd7b3-e6e3-406a-9b25-d8ae8c581e60","Type":"ContainerStarted","Data":"66b5927fbbc4a4a0db884ee5f3a72ed250aba2ac9950f06ce2fbd10953f95100"} Jul 15 20:02:02.664160 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.664144 2568 scope.go:117] "RemoveContainer" containerID="c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa" Jul 15 20:02:02.664420 ip-10-0-132-25 kubenswrapper[2568]: E0715 20:02:02.664397 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa\": container with ID starting with c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa not found: ID does not exist" containerID="c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa" Jul 15 20:02:02.664420 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.664425 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa"} err="failed to get container status \"c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa\": rpc error: code = NotFound desc = could not find container \"c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa\": container with ID starting with c4e5213a1d1322a7d383d0d6e90ba08aa25a97b54cc02fe5dbfb24ce10d93ffa not found: ID does not exist" Jul 15 20:02:02.664592 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.664442 2568 scope.go:117] "RemoveContainer" containerID="d2884643d1732beda9cbb81cbd48bbf52e7e7b980aeb6dcdecb513a931c80458" Jul 15 20:02:02.683144 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683125 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-model-cache\") pod \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " Jul 15 20:02:02.683234 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683150 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tmp-dir\") pod \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\" (UID: \"0e9cca96-7893-4edd-9f14-e4cdd27c03ca\") " Jul 15 20:02:02.683300 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683289 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:02.683366 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683303 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tbdt\" (UniqueName: \"kubernetes.io/projected/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-kube-api-access-8tbdt\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:02.683366 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683311 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:02.683366 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683319 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:02.683460 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683399 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e9cca96-7893-4edd-9f14-e4cdd27c03ca" (UID: "0e9cca96-7893-4edd-9f14-e4cdd27c03ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:02.683493 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.683455 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e9cca96-7893-4edd-9f14-e4cdd27c03ca" (UID: "0e9cca96-7893-4edd-9f14-e4cdd27c03ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:02.783867 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.783840 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:02.784144 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.783873 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e9cca96-7893-4edd-9f14-e4cdd27c03ca-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:02.983383 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.983338 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr"] Jul 15 20:02:02.985452 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:02.985430 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-77bf75/rate-test-model-77bf75-kserve-b7c7f5f47-728kr"] Jul 15 20:02:03.247722 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.247657 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" podUID="f18af6b0-b524-49be-86b3-7fd8e2787a41" containerName="main" containerID="cri-o://d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41" gracePeriod=45 Jul 15 20:02:03.483721 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.483699 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:02:03.489240 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489216 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-home\") pod \"f18af6b0-b524-49be-86b3-7fd8e2787a41\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " Jul 15 20:02:03.489354 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489266 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-model-cache\") pod \"f18af6b0-b524-49be-86b3-7fd8e2787a41\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " Jul 15 20:02:03.489354 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489292 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd72s\" (UniqueName: \"kubernetes.io/projected/f18af6b0-b524-49be-86b3-7fd8e2787a41-kube-api-access-pd72s\") pod \"f18af6b0-b524-49be-86b3-7fd8e2787a41\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " Jul 15 20:02:03.489354 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489330 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-tmp-dir\") pod \"f18af6b0-b524-49be-86b3-7fd8e2787a41\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " Jul 15 20:02:03.489575 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489474 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18af6b0-b524-49be-86b3-7fd8e2787a41-tls-certs\") pod \"f18af6b0-b524-49be-86b3-7fd8e2787a41\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " Jul 15 20:02:03.489575 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489472 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-home" (OuterVolumeSpecName: "home") pod "f18af6b0-b524-49be-86b3-7fd8e2787a41" (UID: "f18af6b0-b524-49be-86b3-7fd8e2787a41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:03.489575 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489527 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-dshm\") pod \"f18af6b0-b524-49be-86b3-7fd8e2787a41\" (UID: \"f18af6b0-b524-49be-86b3-7fd8e2787a41\") " Jul 15 20:02:03.489575 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489536 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-model-cache" (OuterVolumeSpecName: "model-cache") pod "f18af6b0-b524-49be-86b3-7fd8e2787a41" (UID: "f18af6b0-b524-49be-86b3-7fd8e2787a41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:03.489725 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489604 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f18af6b0-b524-49be-86b3-7fd8e2787a41" (UID: "f18af6b0-b524-49be-86b3-7fd8e2787a41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:03.489831 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489811 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-model-cache\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:03.489831 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489831 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-tmp-dir\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:03.490004 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.489839 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-home\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:03.491481 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.491457 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18af6b0-b524-49be-86b3-7fd8e2787a41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f18af6b0-b524-49be-86b3-7fd8e2787a41" (UID: "f18af6b0-b524-49be-86b3-7fd8e2787a41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 20:02:03.491481 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.491474 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18af6b0-b524-49be-86b3-7fd8e2787a41-kube-api-access-pd72s" (OuterVolumeSpecName: "kube-api-access-pd72s") pod "f18af6b0-b524-49be-86b3-7fd8e2787a41" (UID: "f18af6b0-b524-49be-86b3-7fd8e2787a41"). InnerVolumeSpecName "kube-api-access-pd72s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:02:03.491605 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.491460 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-dshm" (OuterVolumeSpecName: "dshm") pod "f18af6b0-b524-49be-86b3-7fd8e2787a41" (UID: "f18af6b0-b524-49be-86b3-7fd8e2787a41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 20:02:03.590622 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.590527 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd72s\" (UniqueName: \"kubernetes.io/projected/f18af6b0-b524-49be-86b3-7fd8e2787a41-kube-api-access-pd72s\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:03.590622 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.590560 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f18af6b0-b524-49be-86b3-7fd8e2787a41-tls-certs\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:03.590622 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.590575 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f18af6b0-b524-49be-86b3-7fd8e2787a41-dshm\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:03.665179 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.665148 2568 generic.go:356] "Generic (PLEG): container finished" podID="17fbd7b3-e6e3-406a-9b25-d8ae8c581e60" containerID="97c649de61364e8f14ec9e6d98e381939f4ebb932a8387573c49a4d987531417" exitCode=0 Jul 15 20:02:03.665374 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.665235 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952708128" event={"ID":"17fbd7b3-e6e3-406a-9b25-d8ae8c581e60","Type":"ContainerDied","Data":"97c649de61364e8f14ec9e6d98e381939f4ebb932a8387573c49a4d987531417"} Jul 15 20:02:03.666364 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.666321 2568 generic.go:356] "Generic (PLEG): container finished" podID="f18af6b0-b524-49be-86b3-7fd8e2787a41" containerID="d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41" exitCode=0 Jul 15 20:02:03.666488 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.666376 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" Jul 15 20:02:03.666488 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.666378 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" event={"ID":"f18af6b0-b524-49be-86b3-7fd8e2787a41","Type":"ContainerDied","Data":"d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41"} Jul 15 20:02:03.666488 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.666412 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks" event={"ID":"f18af6b0-b524-49be-86b3-7fd8e2787a41","Type":"ContainerDied","Data":"35f5309152eefa76da84cb1306e908570a7862d25b029737cf42ee5676fc70cc"} Jul 15 20:02:03.666488 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.666430 2568 scope.go:117] "RemoveContainer" containerID="d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41" Jul 15 20:02:03.675121 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.675101 2568 scope.go:117] "RemoveContainer" containerID="d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41" Jul 15 20:02:03.675375 ip-10-0-132-25 kubenswrapper[2568]: E0715 20:02:03.675355 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41\": container with ID starting with d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41 not found: ID does not exist" containerID="d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41" Jul 15 20:02:03.675441 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.675388 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41"} err="failed to get container status \"d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41\": rpc error: code = NotFound desc = could not find container \"d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41\": container with ID starting with d187ea2e2899e5ee459ccccd9b154b7d4409fe665f35cc381566b722f8dddf41 not found: ID does not exist" Jul 15 20:02:03.707923 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.707896 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks"] Jul 15 20:02:03.711282 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:03.711259 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fcd095/rate-test-model-fcd095-kserve-6c74f8c6c5-vfpks"] Jul 15 20:02:04.232809 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.232777 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9cca96-7893-4edd-9f14-e4cdd27c03ca" path="/var/lib/kubelet/pods/0e9cca96-7893-4edd-9f14-e4cdd27c03ca/volumes" Jul 15 20:02:04.233168 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.233128 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18af6b0-b524-49be-86b3-7fd8e2787a41" path="/var/lib/kubelet/pods/f18af6b0-b524-49be-86b3-7fd8e2787a41/volumes" Jul 15 20:02:04.804629 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.804605 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:04.836643 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.836616 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140644952708128_17fbd7b3-e6e3-406a-9b25-d8ae8c581e60/test-curl-7442-140644952708128/0.log" Jul 15 20:02:04.873407 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.873372 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:04.875493 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.875471 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:04.901010 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.900974 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nvwv\" (UniqueName: \"kubernetes.io/projected/17fbd7b3-e6e3-406a-9b25-d8ae8c581e60-kube-api-access-8nvwv\") pod \"17fbd7b3-e6e3-406a-9b25-d8ae8c581e60\" (UID: \"17fbd7b3-e6e3-406a-9b25-d8ae8c581e60\") " Jul 15 20:02:04.903080 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:04.903055 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fbd7b3-e6e3-406a-9b25-d8ae8c581e60-kube-api-access-8nvwv" (OuterVolumeSpecName: "kube-api-access-8nvwv") pod "17fbd7b3-e6e3-406a-9b25-d8ae8c581e60" (UID: "17fbd7b3-e6e3-406a-9b25-d8ae8c581e60"). InnerVolumeSpecName "kube-api-access-8nvwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:02:05.001698 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.001660 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nvwv\" (UniqueName: \"kubernetes.io/projected/17fbd7b3-e6e3-406a-9b25-d8ae8c581e60-kube-api-access-8nvwv\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:05.017250 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017222 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:05.017542 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017528 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fbd7b3-e6e3-406a-9b25-d8ae8c581e60" containerName="test-curl-7442-140644952708128" Jul 15 20:02:05.017586 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017544 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fbd7b3-e6e3-406a-9b25-d8ae8c581e60" containerName="test-curl-7442-140644952708128" Jul 15 20:02:05.017586 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017564 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9cca96-7893-4edd-9f14-e4cdd27c03ca" containerName="main" Jul 15 20:02:05.017586 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017569 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9cca96-7893-4edd-9f14-e4cdd27c03ca" containerName="main" Jul 15 20:02:05.017586 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017576 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18af6b0-b524-49be-86b3-7fd8e2787a41" containerName="main" Jul 15 20:02:05.017586 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017581 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18af6b0-b524-49be-86b3-7fd8e2787a41" containerName="main" Jul 15 20:02:05.017735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017633 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18af6b0-b524-49be-86b3-7fd8e2787a41" containerName="main" Jul 15 20:02:05.017735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017639 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9cca96-7893-4edd-9f14-e4cdd27c03ca" containerName="main" Jul 15 20:02:05.017735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.017645 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fbd7b3-e6e3-406a-9b25-d8ae8c581e60" containerName="test-curl-7442-140644952708128" Jul 15 20:02:05.020432 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.020411 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:05.033317 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.033292 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:05.102708 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.102671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmms\" (UniqueName: \"kubernetes.io/projected/e99267b1-744c-4c31-9f78-91a4bd9d692f-kube-api-access-8vmms\") pod \"test-curl-7442-140644952708128\" (UID: \"e99267b1-744c-4c31-9f78-91a4bd9d692f\") " pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:05.204001 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.203949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmms\" (UniqueName: \"kubernetes.io/projected/e99267b1-744c-4c31-9f78-91a4bd9d692f-kube-api-access-8vmms\") pod \"test-curl-7442-140644952708128\" (UID: \"e99267b1-744c-4c31-9f78-91a4bd9d692f\") " pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:05.220287 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.220253 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmms\" (UniqueName: \"kubernetes.io/projected/e99267b1-744c-4c31-9f78-91a4bd9d692f-kube-api-access-8vmms\") pod \"test-curl-7442-140644952708128\" (UID: \"e99267b1-744c-4c31-9f78-91a4bd9d692f\") " pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:05.329272 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.329188 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:05.463273 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.463248 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:05.464607 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:02:05.464582 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode99267b1_744c_4c31_9f78_91a4bd9d692f.slice/crio-8ae1df740fa124a2623076a4c70475d39e8b51b684590c8424cb51a089cda496 WatchSource:0}: Error finding container 8ae1df740fa124a2623076a4c70475d39e8b51b684590c8424cb51a089cda496: Status 404 returned error can't find the container with id 8ae1df740fa124a2623076a4c70475d39e8b51b684590c8424cb51a089cda496 Jul 15 20:02:05.679146 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.679061 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:05.679146 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.679068 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66b5927fbbc4a4a0db884ee5f3a72ed250aba2ac9950f06ce2fbd10953f95100" Jul 15 20:02:05.680237 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.680207 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952708128" event={"ID":"e99267b1-744c-4c31-9f78-91a4bd9d692f","Type":"ContainerStarted","Data":"8ae1df740fa124a2623076a4c70475d39e8b51b684590c8424cb51a089cda496"} Jul 15 20:02:05.693307 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:05.693283 2568 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7442-140644952708128" oldPodUID="17fbd7b3-e6e3-406a-9b25-d8ae8c581e60" podUID="e99267b1-744c-4c31-9f78-91a4bd9d692f" Jul 15 20:02:06.228153 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:06.228123 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fbd7b3-e6e3-406a-9b25-d8ae8c581e60" path="/var/lib/kubelet/pods/17fbd7b3-e6e3-406a-9b25-d8ae8c581e60/volumes" Jul 15 20:02:06.686835 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:06.686802 2568 generic.go:356] "Generic (PLEG): container finished" podID="e99267b1-744c-4c31-9f78-91a4bd9d692f" containerID="d4c47993c3e460a82fc9e3dce9206ed98c48a037e08e8f54515bebe1c74e6b86" exitCode=0 Jul 15 20:02:06.687223 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:06.686857 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7442-140644952708128" event={"ID":"e99267b1-744c-4c31-9f78-91a4bd9d692f","Type":"ContainerDied","Data":"d4c47993c3e460a82fc9e3dce9206ed98c48a037e08e8f54515bebe1c74e6b86"} Jul 15 20:02:07.817054 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:07.817030 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:07.825543 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:07.825522 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vmms\" (UniqueName: \"kubernetes.io/projected/e99267b1-744c-4c31-9f78-91a4bd9d692f-kube-api-access-8vmms\") pod \"e99267b1-744c-4c31-9f78-91a4bd9d692f\" (UID: \"e99267b1-744c-4c31-9f78-91a4bd9d692f\") " Jul 15 20:02:07.827526 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:07.827493 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99267b1-744c-4c31-9f78-91a4bd9d692f-kube-api-access-8vmms" (OuterVolumeSpecName: "kube-api-access-8vmms") pod "e99267b1-744c-4c31-9f78-91a4bd9d692f" (UID: "e99267b1-744c-4c31-9f78-91a4bd9d692f"). InnerVolumeSpecName "kube-api-access-8vmms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 20:02:07.857096 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:07.857069 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7442-140644952708128_e99267b1-744c-4c31-9f78-91a4bd9d692f/test-curl-7442-140644952708128/0.log" Jul 15 20:02:07.896118 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:07.896087 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:07.907103 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:07.907077 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7442-140644952708128"] Jul 15 20:02:07.926818 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:07.926797 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmms\" (UniqueName: \"kubernetes.io/projected/e99267b1-744c-4c31-9f78-91a4bd9d692f-kube-api-access-8vmms\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:08.121362 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.121312 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp"] Jul 15 20:02:08.121766 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.121751 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99267b1-744c-4c31-9f78-91a4bd9d692f" containerName="test-curl-7442-140644952708128" Jul 15 20:02:08.121806 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.121770 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99267b1-744c-4c31-9f78-91a4bd9d692f" containerName="test-curl-7442-140644952708128" Jul 15 20:02:08.121854 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.121835 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99267b1-744c-4c31-9f78-91a4bd9d692f" containerName="test-curl-7442-140644952708128" Jul 15 20:02:08.125009 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.124993 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" Jul 15 20:02:08.127863 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.127844 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhrw8\"" Jul 15 20:02:08.128434 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.128410 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:02:08.134667 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.134643 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp"] Jul 15 20:02:08.228691 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.228661 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e99267b1-744c-4c31-9f78-91a4bd9d692f" path="/var/lib/kubelet/pods/e99267b1-744c-4c31-9f78-91a4bd9d692f/volumes" Jul 15 20:02:08.229280 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.229256 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp\" (UID: \"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" Jul 15 20:02:08.329891 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.329857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp\" (UID: \"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" Jul 15 20:02:08.330500 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.330480 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp\" (UID: \"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" Jul 15 20:02:08.435722 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.435639 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" Jul 15 20:02:08.576481 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.576391 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp"] Jul 15 20:02:08.578584 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:02:08.578554 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c6a8c4_33c4_4e4a_b3ca_b72e4c5c85dc.slice/crio-caadcd89394600a12c6cd68b8f3db647052abca3f31d010cf919f2e7326173b9 WatchSource:0}: Error finding container caadcd89394600a12c6cd68b8f3db647052abca3f31d010cf919f2e7326173b9: Status 404 returned error can't find the container with id caadcd89394600a12c6cd68b8f3db647052abca3f31d010cf919f2e7326173b9 Jul 15 20:02:08.702793 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.702696 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" event={"ID":"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc","Type":"ContainerStarted","Data":"caadcd89394600a12c6cd68b8f3db647052abca3f31d010cf919f2e7326173b9"} Jul 15 20:02:08.704172 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.704147 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7442-140644952708128" Jul 15 20:02:08.704172 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:08.704131 2568 scope.go:117] "RemoveContainer" containerID="d4c47993c3e460a82fc9e3dce9206ed98c48a037e08e8f54515bebe1c74e6b86" Jul 15 20:02:09.048399 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:09.048358 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:09.145924 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:09.145885 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:09.712380 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:09.712267 2568 generic.go:356] "Generic (PLEG): container finished" podID="69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc" containerID="a22217e8b2bf214460f5a155362b15ce589e80febe0130417619bf80d70f509e" exitCode=0 Jul 15 20:02:09.712380 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:09.712368 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" event={"ID":"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc","Type":"ContainerDied","Data":"a22217e8b2bf214460f5a155362b15ce589e80febe0130417619bf80d70f509e"} Jul 15 20:02:10.850440 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:10.850404 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" Jul 15 20:02:10.955537 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:10.955501 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc-maas-api-service-ca\") pod \"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc\" (UID: \"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc\") " Jul 15 20:02:10.955868 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:10.955846 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc" (UID: "69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:02:11.056160 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:11.056123 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc-maas-api-service-ca\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:11.723549 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:11.723511 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" Jul 15 20:02:11.723549 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:11.723541 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp" event={"ID":"69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc","Type":"ContainerDied","Data":"caadcd89394600a12c6cd68b8f3db647052abca3f31d010cf919f2e7326173b9"} Jul 15 20:02:11.723790 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:11.723588 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="caadcd89394600a12c6cd68b8f3db647052abca3f31d010cf919f2e7326173b9" Jul 15 20:02:18.963501 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:18.963407 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:22.436964 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:22.436925 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:02:23.310176 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:23.310137 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:30.608221 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:30.608187 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:30.708506 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:30.708472 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:34.137938 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.137899 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:34.374605 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.374565 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:34.521233 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.521198 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7"] Jul 15 20:02:34.521529 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.521515 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc" containerName="revoke-keys" Jul 15 20:02:34.521529 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.521529 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc" containerName="revoke-keys" Jul 15 20:02:34.521621 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.521580 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc" containerName="revoke-keys" Jul 15 20:02:34.526814 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.526794 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" Jul 15 20:02:34.532371 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.532339 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qhrw8\"" Jul 15 20:02:34.532509 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.532384 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 20:02:34.543159 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.543135 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7"] Jul 15 20:02:34.658841 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.658807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86c457-13b9-4510-a85e-cf8ff33aaccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7\" (UID: \"6c86c457-13b9-4510-a85e-cf8ff33aaccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" Jul 15 20:02:34.760122 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.760084 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86c457-13b9-4510-a85e-cf8ff33aaccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7\" (UID: \"6c86c457-13b9-4510-a85e-cf8ff33aaccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" Jul 15 20:02:34.760763 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.760744 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86c457-13b9-4510-a85e-cf8ff33aaccb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7\" (UID: \"6c86c457-13b9-4510-a85e-cf8ff33aaccb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" Jul 15 20:02:34.837002 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.836912 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" Jul 15 20:02:34.976891 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:34.976867 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7"] Jul 15 20:02:34.978854 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:02:34.978823 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c86c457_13b9_4510_a85e_cf8ff33aaccb.slice/crio-096e6c562290ebbef31b2d2a3639d77b07b95b592e1fe56044bfae984f10fd4b WatchSource:0}: Error finding container 096e6c562290ebbef31b2d2a3639d77b07b95b592e1fe56044bfae984f10fd4b: Status 404 returned error can't find the container with id 096e6c562290ebbef31b2d2a3639d77b07b95b592e1fe56044bfae984f10fd4b Jul 15 20:02:35.416945 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:35.416853 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:35.515984 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:35.515938 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:35.846670 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:35.844787 2568 generic.go:356] "Generic (PLEG): container finished" podID="6c86c457-13b9-4510-a85e-cf8ff33aaccb" containerID="a21ba163a74d868c8648fae3268cd4d6e185149544517a7bb0a082b09dfd73e4" exitCode=0 Jul 15 20:02:35.846670 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:35.844855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" event={"ID":"6c86c457-13b9-4510-a85e-cf8ff33aaccb","Type":"ContainerDied","Data":"a21ba163a74d868c8648fae3268cd4d6e185149544517a7bb0a082b09dfd73e4"} Jul 15 20:02:35.846670 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:35.844936 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" event={"ID":"6c86c457-13b9-4510-a85e-cf8ff33aaccb","Type":"ContainerStarted","Data":"096e6c562290ebbef31b2d2a3639d77b07b95b592e1fe56044bfae984f10fd4b"} Jul 15 20:02:36.979259 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:36.979229 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" Jul 15 20:02:37.082465 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:37.082429 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86c457-13b9-4510-a85e-cf8ff33aaccb-maas-api-service-ca\") pod \"6c86c457-13b9-4510-a85e-cf8ff33aaccb\" (UID: \"6c86c457-13b9-4510-a85e-cf8ff33aaccb\") " Jul 15 20:02:37.082824 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:37.082799 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c86c457-13b9-4510-a85e-cf8ff33aaccb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c86c457-13b9-4510-a85e-cf8ff33aaccb" (UID: "6c86c457-13b9-4510-a85e-cf8ff33aaccb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 20:02:37.183496 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:37.183408 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c86c457-13b9-4510-a85e-cf8ff33aaccb-maas-api-service-ca\") on node \"ip-10-0-132-25.ec2.internal\" DevicePath \"\"" Jul 15 20:02:37.856848 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:37.856820 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" Jul 15 20:02:37.857037 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:37.856869 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7" event={"ID":"6c86c457-13b9-4510-a85e-cf8ff33aaccb","Type":"ContainerDied","Data":"096e6c562290ebbef31b2d2a3639d77b07b95b592e1fe56044bfae984f10fd4b"} Jul 15 20:02:37.857037 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:37.856915 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="096e6c562290ebbef31b2d2a3639d77b07b95b592e1fe56044bfae984f10fd4b" Jul 15 20:02:42.769856 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:42.769814 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:42.869800 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:42.869763 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:45.586426 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:45.586381 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:46.994538 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:46.994503 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:49.007907 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:49.007871 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l2vjt"] Jul 15 20:02:49.649973 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:49.649934 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:57.275364 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:57.275310 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:02:57.370585 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:02:57.370554 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:00.487847 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:00.487816 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:00.722709 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:00.722682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:08.101164 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:08.101139 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d9998b69-r69fd_78359574-4e48-4da2-84b4-a3f0269e58a5/manager/0.log" Jul 15 20:03:08.112746 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:08.112718 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d9998b69-r69fd_78359574-4e48-4da2-84b4-a3f0269e58a5/copy-manifests/0.log" Jul 15 20:03:09.384509 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:09.384480 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-ee1301-8fnfs_3cb38451-2c44-4153-9ff7-e8e2c98d3846/revoke-keys/0.log" Jul 15 20:03:10.354558 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:10.354524 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-77bf75-mgbmp_69c6a8c4-33c4-4e4a-b3ca-b72e4c5c85dc/revoke-keys/0.log" Jul 15 20:03:10.475041 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:10.475016 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-fcd095-nqjv7_6c86c457-13b9-4510-a85e-cf8ff33aaccb/revoke-keys/0.log" Jul 15 20:03:11.457824 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.457795 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv_2db38e70-b464-4432-9d6c-565774c67c79/util/0.log" Jul 15 20:03:11.473655 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.473618 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv_2db38e70-b464-4432-9d6c-565774c67c79/pull/0.log" Jul 15 20:03:11.498208 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.498181 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv_2db38e70-b464-4432-9d6c-565774c67c79/extract/0.log" Jul 15 20:03:11.622641 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.622607 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm_d59775d5-69d8-4981-864c-c83b05a4d665/pull/0.log" Jul 15 20:03:11.632740 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.632718 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm_d59775d5-69d8-4981-864c-c83b05a4d665/extract/0.log" Jul 15 20:03:11.645870 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.645843 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm_d59775d5-69d8-4981-864c-c83b05a4d665/util/0.log" Jul 15 20:03:11.767829 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.767803 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp_47924db6-c526-4ee9-9124-6eb598ea17f5/util/0.log" Jul 15 20:03:11.784123 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.784094 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp_47924db6-c526-4ee9-9124-6eb598ea17f5/pull/0.log" Jul 15 20:03:11.794735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.794710 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp_47924db6-c526-4ee9-9124-6eb598ea17f5/extract/0.log" Jul 15 20:03:11.913965 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.913937 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb_cd4bbd29-ab05-4e10-817b-5acd62aeb8f3/util/0.log" Jul 15 20:03:11.926483 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.926446 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb_cd4bbd29-ab05-4e10-817b-5acd62aeb8f3/pull/0.log" Jul 15 20:03:11.942047 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:11.942023 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb_cd4bbd29-ab05-4e10-817b-5acd62aeb8f3/extract/0.log" Jul 15 20:03:12.686125 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:12.686097 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rvzp5_5b89624c-25c9-4517-a5ed-0e455ca50ad3/kuadrant-console-plugin/0.log" Jul 15 20:03:13.079539 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:13.079509 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l2vjt_e9c8c178-d752-4caa-b653-c549fb595ef4/limitador/0.log" Jul 15 20:03:13.837104 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:13.837077 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c95658bf4-76mh2_f89ecfac-1ba1-4152-93c3-b97f9d27175e/kube-auth-proxy/0.log" Jul 15 20:03:15.432322 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:15.432292 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-gwwdj_50c03f19-0056-48ca-81e8-5f05fe0f1b79/main/0.log" Jul 15 20:03:21.590091 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.590044 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4t46/must-gather-tskzx"] Jul 15 20:03:21.590507 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.590372 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c86c457-13b9-4510-a85e-cf8ff33aaccb" containerName="revoke-keys" Jul 15 20:03:21.590507 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.590383 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c86c457-13b9-4510-a85e-cf8ff33aaccb" containerName="revoke-keys" Jul 15 20:03:21.590507 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.590438 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c86c457-13b9-4510-a85e-cf8ff33aaccb" containerName="revoke-keys" Jul 15 20:03:21.592608 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.592576 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:21.593517 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.593500 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:21.597328 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.597309 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4t46\"/\"kube-root-ca.crt\"" Jul 15 20:03:21.597973 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.597957 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g4t46\"/\"default-dockercfg-jjljq\"" Jul 15 20:03:21.600404 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.600384 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4t46\"/\"openshift-service-ca.crt\"" Jul 15 20:03:21.607729 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.607709 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4t46/must-gather-tskzx"] Jul 15 20:03:21.640757 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.640727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfgt\" (UniqueName: \"kubernetes.io/projected/aca923c0-111d-4721-bddc-e6b0b6d72496-kube-api-access-hpfgt\") pod \"must-gather-tskzx\" (UID: \"aca923c0-111d-4721-bddc-e6b0b6d72496\") " pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:21.640882 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.640778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aca923c0-111d-4721-bddc-e6b0b6d72496-must-gather-output\") pod \"must-gather-tskzx\" (UID: \"aca923c0-111d-4721-bddc-e6b0b6d72496\") " pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:21.711048 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.711012 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:21.742129 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.742099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aca923c0-111d-4721-bddc-e6b0b6d72496-must-gather-output\") pod \"must-gather-tskzx\" (UID: \"aca923c0-111d-4721-bddc-e6b0b6d72496\") " pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:21.742300 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.742192 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfgt\" (UniqueName: \"kubernetes.io/projected/aca923c0-111d-4721-bddc-e6b0b6d72496-kube-api-access-hpfgt\") pod \"must-gather-tskzx\" (UID: \"aca923c0-111d-4721-bddc-e6b0b6d72496\") " pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:21.742483 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.742460 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/aca923c0-111d-4721-bddc-e6b0b6d72496-must-gather-output\") pod \"must-gather-tskzx\" (UID: \"aca923c0-111d-4721-bddc-e6b0b6d72496\") " pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:21.758328 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.758298 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfgt\" (UniqueName: \"kubernetes.io/projected/aca923c0-111d-4721-bddc-e6b0b6d72496-kube-api-access-hpfgt\") pod \"must-gather-tskzx\" (UID: \"aca923c0-111d-4721-bddc-e6b0b6d72496\") " pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:21.902983 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:21.902893 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4t46/must-gather-tskzx" Jul 15 20:03:22.055153 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:22.055125 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4t46/must-gather-tskzx"] Jul 15 20:03:22.058540 ip-10-0-132-25 kubenswrapper[2568]: W0715 20:03:22.058503 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaca923c0_111d_4721_bddc_e6b0b6d72496.slice/crio-99d5d493ba31f739992f242356468c6c2d699ac49b64b5e96dd1c2d0ebc97eb7 WatchSource:0}: Error finding container 99d5d493ba31f739992f242356468c6c2d699ac49b64b5e96dd1c2d0ebc97eb7: Status 404 returned error can't find the container with id 99d5d493ba31f739992f242356468c6c2d699ac49b64b5e96dd1c2d0ebc97eb7 Jul 15 20:03:22.086891 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:22.086861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4t46/must-gather-tskzx" event={"ID":"aca923c0-111d-4721-bddc-e6b0b6d72496","Type":"ContainerStarted","Data":"99d5d493ba31f739992f242356468c6c2d699ac49b64b5e96dd1c2d0ebc97eb7"} Jul 15 20:03:23.096927 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:23.096222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4t46/must-gather-tskzx" event={"ID":"aca923c0-111d-4721-bddc-e6b0b6d72496","Type":"ContainerStarted","Data":"209aabdb6e0c9d457721fd0d53bf40c25450dc09b7b0f125b0cf85384d9cc40c"} Jul 15 20:03:23.096927 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:23.096278 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4t46/must-gather-tskzx" event={"ID":"aca923c0-111d-4721-bddc-e6b0b6d72496","Type":"ContainerStarted","Data":"e67cff82947a4350327bb77e098bb84ad4d78f8d61b8fe54af2a4a38544b1496"} Jul 15 20:03:23.124534 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:23.124473 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4t46/must-gather-tskzx" podStartSLOduration=1.523413471 podStartE2EDuration="2.124454713s" totalImagesPullingTime="601.041242ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 20:03:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:03:23.121380878 +0000 UTC m=+2937.312258683" watchObservedRunningTime="2026-07-15 20:03:23.124454713 +0000 UTC m=+2937.315332517" Jul 15 20:03:24.818631 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:24.818591 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-64nd4_6aa8fab3-6ea1-41fd-9ed6-182447618470/global-pull-secret-syncer/0.log" Jul 15 20:03:25.059816 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:25.059777 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" Jul 15 20:03:25.280368 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:25.280314 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-25.ec2.internal_aefba3ae57e796fcd307ce94d0d35ce8/haproxy/0.log" Jul 15 20:03:29.496615 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.496574 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv_2db38e70-b464-4432-9d6c-565774c67c79/extract/0.log" Jul 15 20:03:29.570203 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.570171 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv_2db38e70-b464-4432-9d6c-565774c67c79/util/0.log" Jul 15 20:03:29.631900 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.631868 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759bpxlv_2db38e70-b464-4432-9d6c-565774c67c79/pull/0.log" Jul 15 20:03:29.686076 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.685940 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm_d59775d5-69d8-4981-864c-c83b05a4d665/extract/0.log" Jul 15 20:03:29.739680 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.739594 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm_d59775d5-69d8-4981-864c-c83b05a4d665/util/0.log" Jul 15 20:03:29.784016 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.783939 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0p4zhm_d59775d5-69d8-4981-864c-c83b05a4d665/pull/0.log" Jul 15 20:03:29.843307 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.843272 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp_47924db6-c526-4ee9-9124-6eb598ea17f5/extract/0.log" Jul 15 20:03:29.904104 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.904021 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp_47924db6-c526-4ee9-9124-6eb598ea17f5/util/0.log" Jul 15 20:03:29.950221 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:29.950190 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73857xp_47924db6-c526-4ee9-9124-6eb598ea17f5/pull/0.log" Jul 15 20:03:30.000966 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:30.000929 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb_cd4bbd29-ab05-4e10-817b-5acd62aeb8f3/extract/0.log" Jul 15 20:03:30.063260 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:30.063167 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb_cd4bbd29-ab05-4e10-817b-5acd62aeb8f3/util/0.log" Jul 15 20:03:30.125363 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:30.125314 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1gwxbb_cd4bbd29-ab05-4e10-817b-5acd62aeb8f3/pull/0.log" Jul 15 20:03:30.676662 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:30.676633 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-rvzp5_5b89624c-25c9-4517-a5ed-0e455ca50ad3/kuadrant-console-plugin/0.log" Jul 15 20:03:30.955762 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:30.955668 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l2vjt_e9c8c178-d752-4caa-b653-c549fb595ef4/limitador/0.log" Jul 15 20:03:33.573765 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:33.573734 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvxz9_4fd1a80e-868b-497a-968f-7c4c187954cd/node-exporter/0.log" Jul 15 20:03:33.608893 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:33.608860 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvxz9_4fd1a80e-868b-497a-968f-7c4c187954cd/kube-rbac-proxy/0.log" Jul 15 20:03:33.653304 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:33.653281 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zvxz9_4fd1a80e-868b-497a-968f-7c4c187954cd/init-textfile/0.log" Jul 15 20:03:34.748270 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.748235 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv"] Jul 15 20:03:34.755384 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.755362 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.792105 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.792073 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv"] Jul 15 20:03:34.869869 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.869827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-podres\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.870066 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.869918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2bgr\" (UniqueName: \"kubernetes.io/projected/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-kube-api-access-r2bgr\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.870066 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.869958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-sys\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.870203 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.870063 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-lib-modules\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.870203 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.870099 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-proc\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971480 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971437 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-lib-modules\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971480 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-proc\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-podres\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2bgr\" (UniqueName: \"kubernetes.io/projected/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-kube-api-access-r2bgr\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-proc\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-sys\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971671 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-podres\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971618 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-lib-modules\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.971735 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.971731 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-sys\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:34.988881 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:34.988843 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2bgr\" (UniqueName: \"kubernetes.io/projected/344b29fa-c505-40c5-b5c0-a8b7bde6dfa0-kube-api-access-r2bgr\") pod \"perf-node-gather-daemonset-f8nqv\" (UID: \"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0\") " pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:35.066239 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:35.066160 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:35.235457 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:35.235414 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv"] Jul 15 20:03:36.196063 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:36.196025 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" event={"ID":"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0","Type":"ContainerStarted","Data":"366746a21ae51f67bc2cdf3fd608af3f8455b9dda72c5660bf7952386d167bb4"} Jul 15 20:03:36.196497 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:36.196103 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" event={"ID":"344b29fa-c505-40c5-b5c0-a8b7bde6dfa0","Type":"ContainerStarted","Data":"df6f02cc2cf0e9e07ddc3b41565f36751721218db18df0011259dd8c947e821f"} Jul 15 20:03:36.196497 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:36.196175 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:36.232719 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:36.232658 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" podStartSLOduration=2.23263618 podStartE2EDuration="2.23263618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 20:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 20:03:36.231532404 +0000 UTC m=+2950.422410220" watchObservedRunningTime="2026-07-15 20:03:36.23263618 +0000 UTC m=+2950.423513987" Jul 15 20:03:36.293073 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:36.293044 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-rp5lk_a9db371c-b7b8-448b-b63c-3be20f40bb63/networking-console-plugin/0.log" Jul 15 20:03:37.551986 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:37.551951 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d74dbd896-pctjz_4b109634-10fd-4ae5-8254-3750be26ac57/console/0.log" Jul 15 20:03:39.628049 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:39.628015 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2rggc_f9e5338b-132f-4529-9a5d-8d38e2671420/dns/0.log" Jul 15 20:03:39.667087 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:39.667040 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2rggc_f9e5338b-132f-4529-9a5d-8d38e2671420/kube-rbac-proxy/0.log" Jul 15 20:03:39.915237 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:39.915144 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pwqs8_902c3290-49b2-443b-a898-e758ae8dfa04/dns-node-resolver/0.log" Jul 15 20:03:40.757709 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:40.757678 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f6db4b9f-ktvp5_03bab594-36e8-4d64-8891-b440d7cda702/registry/0.log" Jul 15 20:03:40.803519 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:40.803492 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4r98t_4f396e78-e744-4180-99ab-5e6b831458a1/node-ca/0.log" Jul 15 20:03:42.215681 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:42.215651 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g4t46/perf-node-gather-daemonset-f8nqv" Jul 15 20:03:42.312502 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:42.312465 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c95658bf4-76mh2_f89ecfac-1ba1-4152-93c3-b97f9d27175e/kube-auth-proxy/0.log" Jul 15 20:03:43.859248 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:43.859212 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kkt6d_7f3ca659-d65d-4006-96c9-6de961080e75/serve-healthcheck-canary/0.log" Jul 15 20:03:44.784756 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:44.784718 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-59pm7_c19c9b26-f262-4a96-983b-25677ed981c3/konnectivity-agent/0.log" limit=1024 Jul 15 20:03:44.901978 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:44.901952 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c95658bf4-76mh2_f89ecfac-1ba1-4152-93c3-b97f9d27175e/kube-auth-proxy/0.log" Jul 15 20:03:45.161380 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:45.161336 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xhtm6_c1759d7c-a121-4bc4-9d93-7c9e62deb83b/kube-rbac-proxy/0.log" Jul 15 20:03:45.218302 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:45.218268 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xhtm6_c1759d7c-a121-4bc4-9d93-7c9e62deb83b/exporter/0.log" Jul 15 20:03:45.281103 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:45.281079 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xhtm6_c1759d7c-a121-4bc4-9d93-7c9e62deb83b/extractor/0.log" Jul 15 20:03:46.583317 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:46.583287 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c95658bf4-76mh2_f89ecfac-1ba1-4152-93c3-b97f9d27175e/kube-auth-proxy/0.log" Jul 15 20:03:48.452065 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:48.451978 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d9998b69-r69fd_78359574-4e48-4da2-84b4-a3f0269e58a5/manager/0.log" Jul 15 20:03:48.480162 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:48.480139 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d9998b69-r69fd_78359574-4e48-4da2-84b4-a3f0269e58a5/copy-manifests/0.log" Jul 15 20:03:49.790589 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:49.790563 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-756b7fb4f9-9ddz9_56ce0f04-775a-4021-86be-fb81eb986c8d/manager/0.log" Jul 15 20:03:49.843053 ip-10-0-132-25 kubenswrapper[2568]: I0715 20:03:49.843020 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4lwwf_6223cd64-6bce-41e0-8925-0528c7de52f0/openshift-lws-operator/0.log"