Jul 09 23:26:09.172444 ip-10-0-143-88 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 23:26:09.172458 ip-10-0-143-88 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 23:26:09.172467 ip-10-0-143-88 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 23:26:09.172868 ip-10-0-143-88 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 23:26:19.365418 ip-10-0-143-88 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 23:26:19.365440 ip-10-0-143-88 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ca4d3a69ed494b7cb9f2f30808944aad -- Jul 09 23:27:51.989513 ip-10-0-143-88 systemd[1]: Starting Kubernetes Kubelet... Jul 09 23:27:52.440988 ip-10-0-143-88 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 23:27:52.440988 ip-10-0-143-88 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 23:27:52.440988 ip-10-0-143-88 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 23:27:52.440988 ip-10-0-143-88 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 23:27:52.440988 ip-10-0-143-88 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 23:27:52.444036 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.443947 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 23:27:52.451390 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451371 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 23:27:52.451390 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451389 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 23:27:52.451390 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451393 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451396 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451399 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451402 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451406 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451409 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451412 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451414 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451417 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451419 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451422 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451424 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451427 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451430 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451432 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451435 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451437 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451440 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451443 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 23:27:52.451498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451447 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451451 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451455 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451458 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451460 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451463 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451465 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451467 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451470 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451473 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451475 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451478 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451480 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451483 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451485 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451489 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451492 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451494 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451497 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 23:27:52.451987 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451500 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451504 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451507 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451510 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451512 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451515 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451517 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451521 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451525 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451528 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451530 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451533 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451536 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451539 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451542 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451545 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451548 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451550 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451553 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451555 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 23:27:52.452696 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451558 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451561 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451563 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451565 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451568 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451570 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451573 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451576 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451580 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451582 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451585 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451588 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451591 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451593 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451596 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451599 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451601 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451603 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451606 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451608 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 23:27:52.453230 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451611 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451613 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451616 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451619 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451621 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451624 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451628 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451631 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451634 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451637 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451640 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451643 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451646 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451648 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451650 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451653 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451655 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451658 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 23:27:52.453720 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.451661 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 23:27:52.455372 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455349 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 23:27:52.455372 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455367 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 23:27:52.455372 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455372 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 23:27:52.455372 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455378 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455381 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455385 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455387 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455391 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455394 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455396 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455399 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455402 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455404 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455407 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455410 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455412 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455415 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455417 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455421 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455425 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455428 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455430 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 23:27:52.455543 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455433 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455436 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455438 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455441 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455444 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455447 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455449 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455451 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455454 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455456 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455459 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455462 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455464 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455467 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455469 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455472 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455475 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455477 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455480 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 23:27:52.456013 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455482 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455485 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455487 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455490 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455492 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455494 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455497 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455499 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455503 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455514 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455517 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455520 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455523 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455525 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455527 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455530 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455532 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455534 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455537 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455539 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 23:27:52.456504 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455541 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455544 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455546 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455549 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455551 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455555 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455558 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455560 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455564 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455566 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455569 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455571 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455574 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455576 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455578 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455581 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455583 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455585 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455588 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455590 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 23:27:52.456982 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455593 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455595 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455603 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455605 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455608 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455610 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455613 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455615 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455617 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455620 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455622 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455624 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455626 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455629 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455631 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455634 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455636 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.455638 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456063 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456069 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 23:27:52.457498 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456072 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456076 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456079 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456082 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456086 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456089 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456091 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456094 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456097 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456099 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456102 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456104 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456107 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456109 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456117 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456120 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456122 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456125 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456127 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456129 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 23:27:52.457986 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456132 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456134 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456137 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456139 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456141 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456144 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456146 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456149 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456151 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456153 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456156 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456158 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456161 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456163 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456166 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456169 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456171 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456174 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456176 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456179 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 23:27:52.458512 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456181 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456184 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456186 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456189 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456191 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456194 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456196 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456227 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456230 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456233 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456236 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456239 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456242 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456245 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456247 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456250 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456252 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456255 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456258 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456260 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 23:27:52.458980 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456263 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456265 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456268 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456270 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456273 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456275 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456280 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456284 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456287 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456290 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456292 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456296 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456299 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456302 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456305 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456308 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456310 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456313 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456315 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 23:27:52.459483 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456320 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456328 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456331 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456333 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456335 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456338 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456340 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456354 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456356 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456359 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456362 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456364 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456366 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456369 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456372 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456375 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456377 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.456380 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456455 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456466 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456478 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 23:27:52.459962 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456483 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456487 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456490 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456494 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456499 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456502 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456505 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456509 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456512 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456516 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456519 2562 flags.go:64] FLAG: --cgroup-root="" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456526 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456529 2562 flags.go:64] FLAG: --client-ca-file="" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456538 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456541 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456547 2562 flags.go:64] FLAG: --cluster-domain="" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456549 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456553 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456556 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456559 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456563 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456565 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456570 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 23:27:52.460496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456573 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456576 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456579 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456581 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456584 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456587 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456591 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456594 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456596 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456599 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456602 2562 flags.go:64] FLAG: --enable-server="true" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456604 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456612 2562 flags.go:64] FLAG: --event-burst="100" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456615 2562 flags.go:64] FLAG: --event-qps="50" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456622 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456625 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456627 2562 flags.go:64] FLAG: --eviction-hard="" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456631 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456634 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456637 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456641 2562 flags.go:64] FLAG: --eviction-soft="" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456643 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456646 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456655 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456657 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 23:27:52.461050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456660 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456663 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456666 2562 flags.go:64] FLAG: --feature-gates="" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456669 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456672 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456675 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456679 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456682 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456685 2562 flags.go:64] FLAG: --help="false" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456688 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456691 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456693 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456696 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456699 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456702 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456705 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456708 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456710 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456713 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456716 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456719 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456722 2562 flags.go:64] FLAG: --kube-reserved="" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456725 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456727 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 23:27:52.461678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456730 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456732 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456735 2562 flags.go:64] FLAG: --lock-file="" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456739 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456742 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456744 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456749 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456759 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456762 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456764 2562 flags.go:64] FLAG: --logging-format="text" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456768 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456771 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456773 2562 flags.go:64] FLAG: --manifest-url="" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456776 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456782 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456785 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456789 2562 flags.go:64] FLAG: --max-pods="110" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456792 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456794 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456797 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456800 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456803 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456805 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456808 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456815 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 23:27:52.462367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456818 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456821 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456823 2562 flags.go:64] FLAG: --pod-cidr="" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456826 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456831 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456834 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456837 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456839 2562 flags.go:64] FLAG: --port="10250" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456842 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456845 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cae36738b50d6269" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456849 2562 flags.go:64] FLAG: --qos-reserved="" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456851 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456854 2562 flags.go:64] FLAG: --register-node="true" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456857 2562 flags.go:64] FLAG: --register-with-taints="" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456860 2562 flags.go:64] FLAG: --registry-burst="10" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456868 2562 flags.go:64] FLAG: --registry-qps="5" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456871 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456874 2562 flags.go:64] FLAG: --reserved-memory="" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456877 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456880 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456883 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456886 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456889 2562 flags.go:64] FLAG: --runonce="false" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456892 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456895 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 23:27:52.470510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456897 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456900 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456903 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456906 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456909 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456912 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456914 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456917 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456920 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456922 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456925 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456928 2562 flags.go:64] FLAG: --system-cgroups="" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456930 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456936 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456938 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456941 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456947 2562 flags.go:64] FLAG: --tls-min-version="" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456951 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456954 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456956 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456959 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456962 2562 flags.go:64] FLAG: --v="2" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456965 2562 flags.go:64] FLAG: --version="false" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456977 2562 flags.go:64] FLAG: --vmodule="" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456980 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 23:27:52.471269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.456983 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457112 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457115 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457118 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457122 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457124 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457127 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457129 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457132 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457134 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457137 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457139 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457141 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457144 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457146 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457149 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457151 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457154 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457156 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457158 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 23:27:52.472293 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457161 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457163 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457166 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457169 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457172 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457175 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457177 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457180 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457182 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457184 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457187 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457190 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457192 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457195 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457213 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457216 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457220 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457222 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457225 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457227 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 23:27:52.472848 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457230 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457232 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457234 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457238 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457242 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457245 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457248 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457250 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457253 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457256 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457259 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457261 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457264 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457266 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457268 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457271 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457274 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457277 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457280 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 23:27:52.473362 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457283 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457286 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457290 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457294 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457297 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457300 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457303 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457305 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457307 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457312 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457315 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457317 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457319 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457322 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457324 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457327 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457329 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457332 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457334 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 23:27:52.473966 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457336 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457339 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457341 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457344 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457346 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457349 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457351 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457354 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457356 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457359 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457362 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457365 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457367 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457370 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457372 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457374 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457376 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457379 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457382 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457385 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 23:27:52.474444 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457387 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.457389 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.457397 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.466258 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.466276 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.466296 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.466301 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466361 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466365 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466369 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466371 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466374 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466377 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466380 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466383 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 23:27:52.475167 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466386 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466388 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466392 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466396 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466398 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466402 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466405 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466408 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466411 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466414 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466416 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466419 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466422 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466424 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466426 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466429 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466431 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466434 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466436 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 23:27:52.475575 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466439 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466441 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466445 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466447 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466450 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466452 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466455 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466457 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466460 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466462 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466465 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466467 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466469 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466472 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466474 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466476 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466479 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466481 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466483 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466486 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 23:27:52.476032 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466489 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466492 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466494 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466497 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466500 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466502 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466505 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466508 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466510 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466513 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466515 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466517 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466520 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466522 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466525 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466528 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466531 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466535 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466538 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 23:27:52.476568 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466541 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466544 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466547 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466549 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466552 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466554 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466556 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466559 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466561 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466564 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466566 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466569 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466571 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466574 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466576 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466579 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466581 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466584 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466586 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466589 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 23:27:52.477074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466591 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466593 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466595 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466598 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466600 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466604 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466608 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466611 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466614 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466617 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466619 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466621 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466623 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.466628 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 23:27:52.477583 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466742 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466746 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466749 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466751 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466754 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466756 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466759 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466762 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466764 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466767 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466769 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466772 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466775 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466779 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466782 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466784 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466786 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466789 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466791 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466794 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 23:27:52.477930 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466796 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466799 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466801 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466803 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466806 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466809 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466811 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466813 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466817 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466819 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466822 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466824 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466826 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466829 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466831 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466834 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466836 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466838 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466841 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 23:27:52.478422 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466843 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466845 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466848 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466850 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466853 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466855 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466858 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466860 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466862 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466865 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466867 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466869 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466872 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466874 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466876 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466879 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466881 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466884 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466887 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466889 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 23:27:52.478876 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466892 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466894 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466897 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466900 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466902 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466905 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466908 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466912 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466915 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466917 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466920 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466922 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466925 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466927 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466929 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466932 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466934 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466937 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466939 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466942 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 23:27:52.479466 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466944 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466946 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466948 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466951 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466953 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466955 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466958 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466960 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466963 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466967 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466969 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466972 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466975 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466977 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466980 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466982 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466985 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466988 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466990 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 23:27:52.479952 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:52.466993 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.466997 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.467109 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.469388 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.470950 2562 server.go:1019] "Starting client certificate rotation" Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.471071 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.471127 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.473906 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 23:27:52.480494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.476976 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 23:27:52.488713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.488695 2562 log.go:25] "Validated CRI v1 runtime API" Jul 09 23:27:52.491641 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.491623 2562 log.go:25] "Validated CRI v1 image API" Jul 09 23:27:52.492429 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.492414 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 23:27:52.494908 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.494884 2562 fs.go:135] Filesystem UUIDs: map[55a525b3-07c8-48ce-88e6-247d81780102:/dev/nvme0n1p4 68a2f8ef-f272-46ae-8e4f-6a32f92396a5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 09 23:27:52.494997 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.494907 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 23:27:52.500321 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.500188 2562 manager.go:216] Machine: {Timestamp:2026-07-09 23:27:52.498587364 +0000 UTC m=+0.379339314 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104352 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d4f7b5ee1e5b14b27fe10746d4c63 SystemUUID:ec2d4f7b-5ee1-e5b1-4b27-fe10746d4c63 BootID:ca4d3a69-ed49-4b7c-b9f2-f30808944aad Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:9b:08:2b:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:9b:08:2b:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:e5:38:2d:c5:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 23:27:52.500321 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.500312 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 23:27:52.500489 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.500434 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 23:27:52.501125 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.501108 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 23:27:52.504467 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.504435 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 23:27:52.504603 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.504470 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-88.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 23:27:52.505275 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.505265 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 23:27:52.505307 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.505279 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 23:27:52.505307 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.505287 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 23:27:52.505307 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.505302 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 23:27:52.506948 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.506937 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 23:27:52.510381 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.510369 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 09 23:27:52.510680 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.510670 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 23:27:52.510723 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.510705 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 09 23:27:52.510723 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.510716 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 23:27:52.510798 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.510730 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 23:27:52.510798 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.510739 2562 kubelet.go:398] "Adding apiserver pod source" Jul 09 23:27:52.510798 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.510747 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 23:27:52.511055 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.511044 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 23:27:52.511088 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.511062 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 23:27:52.513615 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.513595 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 23:27:52.515972 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.515959 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 23:27:52.516037 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.515988 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 23:27:52.516170 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516159 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 23:27:52.516234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516175 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 23:27:52.516234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516181 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 23:27:52.516234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516187 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 23:27:52.516234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516195 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 23:27:52.516234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516214 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 23:27:52.516803 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516792 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 23:27:52.516803 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516803 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 23:27:52.516867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516811 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 23:27:52.516867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516817 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 23:27:52.516867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516825 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 23:27:52.516867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.516833 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 23:27:52.517749 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.517735 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 23:27:52.517805 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.517751 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 23:27:52.518728 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.518703 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-db75s" Jul 09 23:27:52.521194 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.521178 2562 server.go:1268] "Started kubelet" Jul 09 23:27:52.521299 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.521274 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 23:27:52.521357 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.521286 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 23:27:52.521405 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.521358 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 23:27:52.522183 ip-10-0-143-88 systemd[1]: Started Kubernetes Kubelet. Jul 09 23:27:52.523026 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.522875 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 09 23:27:52.523026 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.522980 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 23:27:52.523467 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.523442 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 23:27:52.523970 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.523946 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-88.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 23:27:52.524087 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.524069 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-88.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 23:27:52.526011 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.525991 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-db75s" Jul 09 23:27:52.528644 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.528612 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 23:27:52.530839 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.530822 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 23:27:52.530919 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.530845 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 23:27:52.530919 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.530905 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:52.531165 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.529994 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-88.ec2.internal.18c0c2d4ffeac5c7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-88.ec2.internal,UID:ip-10-0-143-88.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-88.ec2.internal,},FirstTimestamp:2026-07-09 23:27:52.521147847 +0000 UTC m=+0.401899797,LastTimestamp:2026-07-09 23:27:52.521147847 +0000 UTC m=+0.401899797,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-88.ec2.internal,}" Jul 09 23:27:52.531558 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.531539 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 23:27:52.531633 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.531594 2562 factory.go:55] Registering systemd factory Jul 09 23:27:52.531702 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.531647 2562 factory.go:223] Registration of the systemd container factory successfully Jul 09 23:27:52.531702 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.531539 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 23:27:52.531702 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.531701 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 23:27:52.532026 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532012 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 09 23:27:52.532026 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532026 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 09 23:27:52.532215 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532184 2562 factory.go:153] Registering CRI-O factory Jul 09 23:27:52.532267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532221 2562 factory.go:223] Registration of the crio container factory successfully Jul 09 23:27:52.532299 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532279 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 23:27:52.532350 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532304 2562 factory.go:103] Registering Raw factory Jul 09 23:27:52.532350 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532321 2562 manager.go:1196] Started watching for new ooms in manager Jul 09 23:27:52.532757 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.532745 2562 manager.go:349] Starting recovery of all containers Jul 09 23:27:52.541390 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.541232 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 23:27:52.542809 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.542793 2562 manager.go:354] Recovery completed Jul 09 23:27:52.544414 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.544357 2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-88.ec2.internal\" not found" node="ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.544594 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.544573 2562 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 09 23:27:52.547657 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.547644 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 23:27:52.549413 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.549395 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory" Jul 09 23:27:52.549465 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.549427 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 23:27:52.549465 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.549445 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID" Jul 09 23:27:52.550012 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.549998 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 23:27:52.550074 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.550012 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 23:27:52.550074 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.549999 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 23:27:52.550074 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.550032 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 09 23:27:52.550749 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.550732 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory" Jul 09 23:27:52.550813 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.550758 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 23:27:52.550813 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.550768 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID" Jul 09 23:27:52.551435 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.551424 2562 policy_none.go:49] "None policy: Start" Jul 09 23:27:52.551468 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.551439 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 23:27:52.551468 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.551449 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 23:27:52.552746 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.552735 2562 policy_none.go:47] "Start" Jul 09 23:27:52.593611 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.593592 2562 manager.go:337] "Starting Device Plugin manager" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.593659 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.593674 2562 server.go:87] "Starting device plugin registration server" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.593943 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.593957 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.594409 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.594487 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.594495 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.595069 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.595123 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.609968 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.611367 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.611405 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.611427 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.611465 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 23:27:52.620334 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.613664 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 23:27:52.694187 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.694093 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 23:27:52.695279 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.695261 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory" Jul 09 23:27:52.695375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.695297 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 23:27:52.695375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.695312 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID" Jul 09 23:27:52.695375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.695336 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.701660 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.701645 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.701705 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.701669 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-88.ec2.internal\": node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:52.711911 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.711890 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal"] Jul 09 23:27:52.711993 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.711950 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 23:27:52.712238 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.712222 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:52.712738 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.712714 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory" Jul 09 23:27:52.712785 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.712744 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 23:27:52.712785 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.712755 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID" Jul 09 23:27:52.714985 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.714973 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 23:27:52.715090 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715075 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.715130 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715106 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 23:27:52.715660 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715645 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory" Jul 09 23:27:52.715717 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715677 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 23:27:52.715717 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715690 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID" Jul 09 23:27:52.715717 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715666 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory" Jul 09 23:27:52.715807 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715720 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 23:27:52.715807 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.715729 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID" Jul 09 23:27:52.717896 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.717882 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.717952 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.717906 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 23:27:52.718543 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.718528 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientMemory" Jul 09 23:27:52.718620 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.718558 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 23:27:52.718620 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.718569 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeHasSufficientPID" Jul 09 23:27:52.733237 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.733215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/833514e0c442828ea1fb7e0b8882a68d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"833514e0c442828ea1fb7e0b8882a68d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.733302 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.733241 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833514e0c442828ea1fb7e0b8882a68d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"833514e0c442828ea1fb7e0b8882a68d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.733302 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.733256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68d79ca2f41a1fb714c439c75beb926d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-88.ec2.internal\" (UID: \"68d79ca2f41a1fb714c439c75beb926d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.738711 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.738698 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-88.ec2.internal\" not found" node="ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.742162 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.742148 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-88.ec2.internal\" not found" node="ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.812762 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.812736 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:52.833485 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.833465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833514e0c442828ea1fb7e0b8882a68d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"833514e0c442828ea1fb7e0b8882a68d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.833543 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.833491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68d79ca2f41a1fb714c439c75beb926d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-88.ec2.internal\" (UID: \"68d79ca2f41a1fb714c439c75beb926d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.833543 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.833508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/833514e0c442828ea1fb7e0b8882a68d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"833514e0c442828ea1fb7e0b8882a68d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.833618 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.833550 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/833514e0c442828ea1fb7e0b8882a68d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"833514e0c442828ea1fb7e0b8882a68d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.833618 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.833566 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833514e0c442828ea1fb7e0b8882a68d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal\" (UID: \"833514e0c442828ea1fb7e0b8882a68d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.833618 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:52.833580 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/68d79ca2f41a1fb714c439c75beb926d-config\") pod \"kube-apiserver-proxy-ip-10-0-143-88.ec2.internal\" (UID: \"68d79ca2f41a1fb714c439c75beb926d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" Jul 09 23:27:52.913175 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:52.913135 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.014005 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.013941 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.041442 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.041414 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:53.046281 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.046263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" Jul 09 23:27:53.114531 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.114500 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.215021 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.214982 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.315533 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.315453 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.415983 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.415954 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.470456 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.470427 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 23:27:53.471105 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.470557 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 23:27:53.471105 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.470588 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 23:27:53.516932 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.516900 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.531389 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.531350 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 23:22:53 +0000 UTC" deadline="2028-03-17 15:12:08.464823476 +0000 UTC" Jul 09 23:27:53.531389 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.531378 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14799h44m14.933447344s" Jul 09 23:27:53.573617 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.573545 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 23:27:53.607872 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:53.607833 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d79ca2f41a1fb714c439c75beb926d.slice/crio-aa4368628e593e848f5b7be88015768def8988fb970ab3e9686c7e8e3c51ddd8 WatchSource:0}: Error finding container aa4368628e593e848f5b7be88015768def8988fb970ab3e9686c7e8e3c51ddd8: Status 404 returned error can't find the container with id aa4368628e593e848f5b7be88015768def8988fb970ab3e9686c7e8e3c51ddd8 Jul 09 23:27:53.608042 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:53.608030 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833514e0c442828ea1fb7e0b8882a68d.slice/crio-b895c836dfb728c30a3fe84f0fe9f4dd9586b32992749257afb5e0b2014e1a2f WatchSource:0}: Error finding container b895c836dfb728c30a3fe84f0fe9f4dd9586b32992749257afb5e0b2014e1a2f: Status 404 returned error can't find the container with id b895c836dfb728c30a3fe84f0fe9f4dd9586b32992749257afb5e0b2014e1a2f Jul 09 23:27:53.615672 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:53.615658 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 23:27:53.617337 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.617316 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.717717 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.717669 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.818232 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.818187 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:53.918763 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:53.918698 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:54.019475 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.019439 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-88.ec2.internal\" not found" Jul 09 23:27:54.101129 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.101098 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 23:27:54.132020 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.131984 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" Jul 09 23:27:54.143536 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.143499 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 23:27:54.144906 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.144872 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" Jul 09 23:27:54.153872 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.153845 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 23:27:54.448527 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.448358 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 23:27:54.512179 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.512150 2562 apiserver.go:52] "Watching apiserver" Jul 09 23:27:54.520993 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.520961 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 23:27:54.521407 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.521379 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-hvswt","openshift-cluster-node-tuning-operator/tuned-4wljd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal","openshift-multus/network-metrics-daemon-fz2qx","openshift-network-diagnostics/network-check-target-mhsqr","openshift-network-operator/iptables-alerter-ng94m","openshift-ovn-kubernetes/ovnkube-node-jdkzc","kube-system/global-pull-secret-syncer-8gm96","kube-system/konnectivity-agent-8vqmv","kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9","openshift-dns/node-resolver-t59c6","openshift-image-registry/node-ca-g8gb8","openshift-multus/multus-additional-cni-plugins-2xx52"] Jul 09 23:27:54.524324 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.524300 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.526840 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.526813 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 23:27:54.527682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.527647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.528767 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.528518 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 23:27:54.528767 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.528533 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 23:27:54.528767 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.528653 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 23:27:54.528998 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.528847 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fc9c\"" Jul 09 23:27:54.528998 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.528938 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 23:27:54.529107 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.528996 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 23:27:54.530165 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.530143 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 23:27:54.531002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.530532 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.531002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.530550 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4mm5x\"" Jul 09 23:27:54.531002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.530676 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 23:27:54.531994 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.531430 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 23:27:54.531994 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.531761 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 23:27:54.533777 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.533480 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:54.533777 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.533512 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 23:27:54.533777 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.533564 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:27:54.533777 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.533658 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6pxbm\"" Jul 09 23:27:54.534044 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.533818 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 23:27:54.534044 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.533822 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 23:27:54.534044 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.533935 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 23:27:54.534876 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.534858 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 23:27:54.535867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.535847 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:54.536000 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.535976 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:27:54.538265 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.538244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.540929 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.540911 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 23:27:54.541050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.541028 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 23:27:54.542353 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.541170 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fx5jx\"" Jul 09 23:27:54.542462 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.541418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.542691 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.541223 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 23:27:54.542842 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.542823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-slash\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.542894 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.542862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-etc-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.542936 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.542905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-ovn\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.542970 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.542934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-cni-bin\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.542970 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.542964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cdbbd61-cd46-4dea-a427-322f8815faca-host\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.543033 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.542989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cdbbd61-cd46-4dea-a427-322f8815faca-serviceca\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.543033 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543094 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543032 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543094 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-os-release\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543094 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543185 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543107 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:54.543185 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-node-log\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543185 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543309 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543309 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543243 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-log-socket\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543309 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-systemd\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543309 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-cni-netd\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb2jf\" (UniqueName: \"kubernetes.io/projected/d7f4db34-e093-4e81-9bde-54b01feb897f-kube-api-access-lb2jf\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovnkube-script-lib\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543363 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klszk\" (UniqueName: \"kubernetes.io/projected/6cdbbd61-cd46-4dea-a427-322f8815faca-kube-api-access-klszk\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovnkube-config\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-env-overrides\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovn-node-metrics-cert\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-system-cni-dir\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543457 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-systemd-units\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkp9\" (UniqueName: \"kubernetes.io/projected/eb3d9b13-8e9b-4501-883a-a8742ae6068a-kube-api-access-lgkp9\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543508 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-cnibin\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhh9p\" (UniqueName: \"kubernetes.io/projected/ef2cbebe-d114-4c91-b31f-9d2698990faa-kube-api-access-bhh9p\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543586 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-kubelet\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-run-netns\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543651 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-var-lib-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543680 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.543986 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.543975 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 23:27:54.544848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.544813 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.545150 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.545134 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gftfg\"" Jul 09 23:27:54.545433 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.545404 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 23:27:54.545500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.545445 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 23:27:54.547195 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.547177 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.547502 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.547486 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 23:27:54.547759 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.547745 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-h95m7\"" Jul 09 23:27:54.549921 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.549651 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.549921 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.549716 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:27:54.549921 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.549731 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jd2pn\"" Jul 09 23:27:54.549921 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.549758 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 23:27:54.551362 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.551340 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 23:27:54.552747 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.552623 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.554998 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.554972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.555369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.555107 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 23:27:54.555873 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.555852 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 23:27:54.556161 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.556145 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 23:27:54.556253 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.556145 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p6nbh\"" Jul 09 23:27:54.557688 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.557659 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bc2nc\"" Jul 09 23:27:54.557793 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.557725 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 23:27:54.558278 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.558254 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 23:27:54.568952 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.568927 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6gbjc" Jul 09 23:27:54.577631 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.575850 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6gbjc" Jul 09 23:27:54.618367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.618295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" event={"ID":"833514e0c442828ea1fb7e0b8882a68d","Type":"ContainerStarted","Data":"b895c836dfb728c30a3fe84f0fe9f4dd9586b32992749257afb5e0b2014e1a2f"} Jul 09 23:27:54.619723 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.619690 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" event={"ID":"68d79ca2f41a1fb714c439c75beb926d","Type":"ContainerStarted","Data":"aa4368628e593e848f5b7be88015768def8988fb970ab3e9686c7e8e3c51ddd8"} Jul 09 23:27:54.632901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.632878 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 23:27:54.641847 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.641666 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 23:27:54.644097 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-kubernetes\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.644249 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644216 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-kubelet\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.644328 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-run-netns\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.644328 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-var-lib-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.644445 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.644445 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644362 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-socket-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.644445 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.644445 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a72e31df-6cc8-4ab8-88f7-0863b285e03f-host-slash\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.644445 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-slash\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-ovn\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644491 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cdbbd61-cd46-4dea-a427-322f8815faca-host\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d45b59a-6c66-4708-afb4-0efcbf6ca153-hosts-file\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-kubelet\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-conf-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.644690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-multus-certs\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-node-log\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644761 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644777 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3f7c514-15d5-4ff6-a32f-de5e43a59120-cni-binary-copy\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-daemon-config\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b38ad682-bd50-4725-b3c1-821e12e3e15f-dbus\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-systemd\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-cni-netd\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644915 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clq9\" (UniqueName: \"kubernetes.io/projected/8d45b59a-6c66-4708-afb4-0efcbf6ca153-kube-api-access-4clq9\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a72e31df-6cc8-4ab8-88f7-0863b285e03f-iptables-alerter-script\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b040f8e8-3927-4005-a787-52003adb5e78-konnectivity-ca\") pod \"konnectivity-agent-8vqmv\" (UID: \"b040f8e8-3927-4005-a787-52003adb5e78\") " pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.644999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bc9\" (UniqueName: \"kubernetes.io/projected/52e676c6-c4db-4322-8eb4-d9c56bd4843a-kube-api-access-h9bc9\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-cnibin\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.645079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-netns\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645090 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-run\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645116 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovnkube-config\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovn-node-metrics-cert\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frrzg\" (UniqueName: \"kubernetes.io/projected/02c11d85-60a2-4b3b-a979-7db7080e13e3-kube-api-access-frrzg\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkp9\" (UniqueName: \"kubernetes.io/projected/eb3d9b13-8e9b-4501-883a-a8742ae6068a-kube-api-access-lgkp9\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645247 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-cnibin\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645274 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-cnibin\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645510 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.645514 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6cdbbd61-cd46-4dea-a427-322f8815faca-host\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645627 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-ovn\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.645663 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs podName:d7f4db34-e093-4e81-9bde-54b01feb897f nodeName:}" failed. No retries permitted until 2026-07-09 23:27:55.145639094 +0000 UTC m=+3.026391044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs") pod "network-metrics-daemon-fz2qx" (UID: "d7f4db34-e093-4e81-9bde-54b01feb897f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-run-systemd\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.645859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645716 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-cni-netd\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645721 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645754 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-node-log\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-registration-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-var-lib-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645908 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx9mk\" (UniqueName: \"kubernetes.io/projected/a72e31df-6cc8-4ab8-88f7-0863b285e03f-kube-api-access-kx9mk\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-kubelet\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-host\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.645977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-system-cni-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-run-netns\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-hostroot\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646022 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-slash\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646066 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52e676c6-c4db-4322-8eb4-d9c56bd4843a-tmp\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646120 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-cni-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovnkube-config\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-etc-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646219 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-cni-bin\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.646677 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-host-cni-bin\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-etc-openvswitch\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cdbbd61-cd46-4dea-a427-322f8815faca-serviceca\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646364 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646493 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-sys-fs\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysconfig\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-os-release\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646670 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-os-release\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646686 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6cdbbd61-cd46-4dea-a427-322f8815faca-serviceca\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-device-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-cni-multus\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysctl-conf\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646789 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-lib-modules\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646819 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.647426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646843 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d45b59a-6c66-4708-afb4-0efcbf6ca153-tmp-dir\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-socket-dir-parent\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646931 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-var-lib-kubelet\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.646956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-tuned\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-log-socket\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb2jf\" (UniqueName: \"kubernetes.io/projected/d7f4db34-e093-4e81-9bde-54b01feb897f-kube-api-access-lb2jf\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysctl-d\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-log-socket\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-systemd\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-sys\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovnkube-script-lib\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klszk\" (UniqueName: \"kubernetes.io/projected/6cdbbd61-cd46-4dea-a427-322f8815faca-kube-api-access-klszk\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647258 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b38ad682-bd50-4725-b3c1-821e12e3e15f-kubelet-config\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647296 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-env-overrides\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ef2cbebe-d114-4c91-b31f-9d2698990faa-cni-binary-copy\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.648267 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-system-cni-dir\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647347 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-os-release\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647363 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-cni-bin\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-etc-kubernetes\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ef2cbebe-d114-4c91-b31f-9d2698990faa-system-cni-dir\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgwc\" (UniqueName: \"kubernetes.io/projected/c3f7c514-15d5-4ff6-a32f-de5e43a59120-kube-api-access-lkgwc\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-modprobe-d\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647503 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-systemd-units\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhh9p\" (UniqueName: \"kubernetes.io/projected/ef2cbebe-d114-4c91-b31f-9d2698990faa-kube-api-access-bhh9p\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647566 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647613 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-env-overrides\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b040f8e8-3927-4005-a787-52003adb5e78-agent-certs\") pod \"konnectivity-agent-8vqmv\" (UID: \"b040f8e8-3927-4005-a787-52003adb5e78\") " pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-k8s-cni-cncf-io\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb3d9b13-8e9b-4501-883a-a8742ae6068a-systemd-units\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.649060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.647780 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovnkube-script-lib\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.649590 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.649494 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb3d9b13-8e9b-4501-883a-a8742ae6068a-ovn-node-metrics-cert\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.655603 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.655576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkp9\" (UniqueName: \"kubernetes.io/projected/eb3d9b13-8e9b-4501-883a-a8742ae6068a-kube-api-access-lgkp9\") pod \"ovnkube-node-jdkzc\" (UID: \"eb3d9b13-8e9b-4501-883a-a8742ae6068a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.656597 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.656572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb2jf\" (UniqueName: \"kubernetes.io/projected/d7f4db34-e093-4e81-9bde-54b01feb897f-kube-api-access-lb2jf\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:54.657004 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.656982 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klszk\" (UniqueName: \"kubernetes.io/projected/6cdbbd61-cd46-4dea-a427-322f8815faca-kube-api-access-klszk\") pod \"node-ca-g8gb8\" (UID: \"6cdbbd61-cd46-4dea-a427-322f8815faca\") " pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.657943 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.657925 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhh9p\" (UniqueName: \"kubernetes.io/projected/ef2cbebe-d114-4c91-b31f-9d2698990faa-kube-api-access-bhh9p\") pod \"multus-additional-cni-plugins-2xx52\" (UID: \"ef2cbebe-d114-4c91-b31f-9d2698990faa\") " pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.748535 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-kubelet\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.748535 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748488 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-conf-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.748769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748550 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-conf-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.748769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748571 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-kubelet\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.748769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-multus-certs\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.748769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.748769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748692 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3f7c514-15d5-4ff6-a32f-de5e43a59120-cni-binary-copy\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.748769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748713 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-multus-certs\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749021 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.748806 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:54.749021 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.748921 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret podName:b38ad682-bd50-4725-b3c1-821e12e3e15f nodeName:}" failed. No retries permitted until 2026-07-09 23:27:55.248903204 +0000 UTC m=+3.129655144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret") pod "global-pull-secret-syncer-8gm96" (UID: "b38ad682-bd50-4725-b3c1-821e12e3e15f") : object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:54.749251 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.748719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-daemon-config\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749351 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749283 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b38ad682-bd50-4725-b3c1-821e12e3e15f-dbus\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.749410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749318 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clq9\" (UniqueName: \"kubernetes.io/projected/8d45b59a-6c66-4708-afb4-0efcbf6ca153-kube-api-access-4clq9\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.749410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749376 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-daemon-config\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a72e31df-6cc8-4ab8-88f7-0863b285e03f-iptables-alerter-script\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.749552 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b040f8e8-3927-4005-a787-52003adb5e78-konnectivity-ca\") pod \"konnectivity-agent-8vqmv\" (UID: \"b040f8e8-3927-4005-a787-52003adb5e78\") " pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.749552 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bc9\" (UniqueName: \"kubernetes.io/projected/52e676c6-c4db-4322-8eb4-d9c56bd4843a-kube-api-access-h9bc9\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.749552 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-cnibin\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749552 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-netns\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749552 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b38ad682-bd50-4725-b3c1-821e12e3e15f-dbus\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.749552 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-run\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frrzg\" (UniqueName: \"kubernetes.io/projected/02c11d85-60a2-4b3b-a979-7db7080e13e3-kube-api-access-frrzg\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749579 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-cnibin\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-registration-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx9mk\" (UniqueName: \"kubernetes.io/projected/a72e31df-6cc8-4ab8-88f7-0863b285e03f-kube-api-access-kx9mk\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749633 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c3f7c514-15d5-4ff6-a32f-de5e43a59120-cni-binary-copy\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-host\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-run\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-system-cni-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-hostroot\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749740 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-netns\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52e676c6-c4db-4322-8eb4-d9c56bd4843a-tmp\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749784 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-hostroot\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-cni-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749842 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-host\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749855 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-sys-fs\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysconfig\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.749889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749892 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-system-cni-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-device-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749931 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b040f8e8-3927-4005-a787-52003adb5e78-konnectivity-ca\") pod \"konnectivity-agent-8vqmv\" (UID: \"b040f8e8-3927-4005-a787-52003adb5e78\") " pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749948 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-registration-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.749932 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-cni-multus\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-cni-multus\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysctl-conf\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750020 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysconfig\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750009 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-sys-fs\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750037 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-lib-modules\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750065 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-device-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d45b59a-6c66-4708-afb4-0efcbf6ca153-tmp-dir\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-lib-modules\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750126 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-cni-dir\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-socket-dir-parent\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750195 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysctl-conf\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.750834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-var-lib-kubelet\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-tuned\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750250 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-multus-socket-dir-parent\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750259 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysctl-d\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750296 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-var-lib-kubelet\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-systemd\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-sys\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d45b59a-6c66-4708-afb4-0efcbf6ca153-tmp-dir\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750392 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-sysctl-d\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b38ad682-bd50-4725-b3c1-821e12e3e15f-kubelet-config\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b38ad682-bd50-4725-b3c1-821e12e3e15f-kubelet-config\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750432 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-systemd\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-os-release\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-cni-bin\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-sys\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-etc-kubernetes\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgwc\" (UniqueName: \"kubernetes.io/projected/c3f7c514-15d5-4ff6-a32f-de5e43a59120-kube-api-access-lkgwc\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750538 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-os-release\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.751579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-modprobe-d\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750541 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-var-lib-cni-bin\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750567 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-etc-kubernetes\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b040f8e8-3927-4005-a787-52003adb5e78-agent-certs\") pod \"konnectivity-agent-8vqmv\" (UID: \"b040f8e8-3927-4005-a787-52003adb5e78\") " pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750644 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-k8s-cni-cncf-io\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-kubernetes\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-socket-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750719 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a72e31df-6cc8-4ab8-88f7-0863b285e03f-host-slash\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750752 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a72e31df-6cc8-4ab8-88f7-0863b285e03f-iptables-alerter-script\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d45b59a-6c66-4708-afb4-0efcbf6ca153-hosts-file\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750803 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c3f7c514-15d5-4ff6-a32f-de5e43a59120-host-run-k8s-cni-cncf-io\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750770 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-modprobe-d\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d45b59a-6c66-4708-afb4-0efcbf6ca153-hosts-file\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-socket-dir\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750909 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-kubernetes\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.752388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.750961 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a72e31df-6cc8-4ab8-88f7-0863b285e03f-host-slash\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.753186 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.751053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/02c11d85-60a2-4b3b-a979-7db7080e13e3-etc-selinux\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.753186 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.752643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52e676c6-c4db-4322-8eb4-d9c56bd4843a-tmp\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.753186 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.752927 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/52e676c6-c4db-4322-8eb4-d9c56bd4843a-etc-tuned\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.753412 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.753389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b040f8e8-3927-4005-a787-52003adb5e78-agent-certs\") pod \"konnectivity-agent-8vqmv\" (UID: \"b040f8e8-3927-4005-a787-52003adb5e78\") " pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.757941 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.757456 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 23:27:54.757941 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.757483 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 23:27:54.757941 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.757501 2562 projected.go:196] Error preparing data for projected volume kube-api-access-t88d9 for pod openshift-network-diagnostics/network-check-target-mhsqr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:54.757941 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:54.757759 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9 podName:37fa4fdd-fff0-4421-a01b-b8139858c0d4 nodeName:}" failed. No retries permitted until 2026-07-09 23:27:55.257738468 +0000 UTC m=+3.138490407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t88d9" (UniqueName: "kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9") pod "network-check-target-mhsqr" (UID: "37fa4fdd-fff0-4421-a01b-b8139858c0d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:54.758331 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.758078 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clq9\" (UniqueName: \"kubernetes.io/projected/8d45b59a-6c66-4708-afb4-0efcbf6ca153-kube-api-access-4clq9\") pod \"node-resolver-t59c6\" (UID: \"8d45b59a-6c66-4708-afb4-0efcbf6ca153\") " pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:54.758832 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.758806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx9mk\" (UniqueName: \"kubernetes.io/projected/a72e31df-6cc8-4ab8-88f7-0863b285e03f-kube-api-access-kx9mk\") pod \"iptables-alerter-ng94m\" (UID: \"a72e31df-6cc8-4ab8-88f7-0863b285e03f\") " pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.759707 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.759668 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frrzg\" (UniqueName: \"kubernetes.io/projected/02c11d85-60a2-4b3b-a979-7db7080e13e3-kube-api-access-frrzg\") pod \"aws-ebs-csi-driver-node-86qw9\" (UID: \"02c11d85-60a2-4b3b-a979-7db7080e13e3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.759996 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.759975 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgwc\" (UniqueName: \"kubernetes.io/projected/c3f7c514-15d5-4ff6-a32f-de5e43a59120-kube-api-access-lkgwc\") pod \"multus-hvswt\" (UID: \"c3f7c514-15d5-4ff6-a32f-de5e43a59120\") " pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.760082 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.760023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bc9\" (UniqueName: \"kubernetes.io/projected/52e676c6-c4db-4322-8eb4-d9c56bd4843a-kube-api-access-h9bc9\") pod \"tuned-4wljd\" (UID: \"52e676c6-c4db-4322-8eb4-d9c56bd4843a\") " pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.841302 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.841268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:27:54.850045 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.850017 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8gb8" Jul 09 23:27:54.858975 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.858953 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2xx52" Jul 09 23:27:54.864795 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.864776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ng94m" Jul 09 23:27:54.871754 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.871725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:27:54.883578 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.883554 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hvswt" Jul 09 23:27:54.889562 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.889538 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wljd" Jul 09 23:27:54.898427 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.898399 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" Jul 09 23:27:54.905491 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:54.905467 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-t59c6" Jul 09 23:27:55.154002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:55.153914 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:55.154161 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.154051 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:55.154161 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.154133 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs podName:d7f4db34-e093-4e81-9bde-54b01feb897f nodeName:}" failed. No retries permitted until 2026-07-09 23:27:56.154097495 +0000 UTC m=+4.034849439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs") pod "network-metrics-daemon-fz2qx" (UID: "d7f4db34-e093-4e81-9bde-54b01feb897f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:55.254882 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:55.254846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:55.255035 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.254997 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:55.255081 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.255071 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret podName:b38ad682-bd50-4725-b3c1-821e12e3e15f nodeName:}" failed. No retries permitted until 2026-07-09 23:27:56.255047257 +0000 UTC m=+4.135799197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret") pod "global-pull-secret-syncer-8gm96" (UID: "b38ad682-bd50-4725-b3c1-821e12e3e15f") : object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:55.356292 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:55.356255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:55.356466 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.356402 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 23:27:55.356466 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.356420 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 23:27:55.356466 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.356429 2562 projected.go:196] Error preparing data for projected volume kube-api-access-t88d9 for pod openshift-network-diagnostics/network-check-target-mhsqr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:55.356583 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:55.356486 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9 podName:37fa4fdd-fff0-4421-a01b-b8139858c0d4 nodeName:}" failed. No retries permitted until 2026-07-09 23:27:56.356468966 +0000 UTC m=+4.237220907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t88d9" (UniqueName: "kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9") pod "network-check-target-mhsqr" (UID: "37fa4fdd-fff0-4421-a01b-b8139858c0d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:55.580312 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:55.580261 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 23:22:55 +0000 UTC" deadline="2028-01-11 19:28:29.140509989 +0000 UTC" Jul 09 23:27:55.580312 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:55.580308 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13220h0m33.560207879s" Jul 09 23:27:55.679832 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.679600 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdbbd61_cd46_4dea_a427_322f8815faca.slice/crio-3c569acd560960fb1026bfd744cfe25d3be273eb6f46aee2070a0d56e53257be WatchSource:0}: Error finding container 3c569acd560960fb1026bfd744cfe25d3be273eb6f46aee2070a0d56e53257be: Status 404 returned error can't find the container with id 3c569acd560960fb1026bfd744cfe25d3be273eb6f46aee2070a0d56e53257be Jul 09 23:27:55.681416 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.681367 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c11d85_60a2_4b3b_a979_7db7080e13e3.slice/crio-77ab776b49678cbbd91f2b06adfc790cf5338133a34a990f5d05aeff692b97d7 WatchSource:0}: Error finding container 77ab776b49678cbbd91f2b06adfc790cf5338133a34a990f5d05aeff692b97d7: Status 404 returned error can't find the container with id 77ab776b49678cbbd91f2b06adfc790cf5338133a34a990f5d05aeff692b97d7 Jul 09 23:27:55.682532 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.682244 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3d9b13_8e9b_4501_883a_a8742ae6068a.slice/crio-d9abd26b657e726b26e289d220a4dd2f76a1c102c6963cfd3f75d102edc431f4 WatchSource:0}: Error finding container d9abd26b657e726b26e289d220a4dd2f76a1c102c6963cfd3f75d102edc431f4: Status 404 returned error can't find the container with id d9abd26b657e726b26e289d220a4dd2f76a1c102c6963cfd3f75d102edc431f4 Jul 09 23:27:55.683175 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.683147 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f7c514_15d5_4ff6_a32f_de5e43a59120.slice/crio-8a737298ea255f5f5ede16b575bc02f60f82a3a94dc1f748c484fc76f4284515 WatchSource:0}: Error finding container 8a737298ea255f5f5ede16b575bc02f60f82a3a94dc1f748c484fc76f4284515: Status 404 returned error can't find the container with id 8a737298ea255f5f5ede16b575bc02f60f82a3a94dc1f748c484fc76f4284515 Jul 09 23:27:55.709873 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.709850 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb040f8e8_3927_4005_a787_52003adb5e78.slice/crio-93e88cd5806cd45d635fb879e9fdbf90d47b68274ae6a9cb009a6037f73da76d WatchSource:0}: Error finding container 93e88cd5806cd45d635fb879e9fdbf90d47b68274ae6a9cb009a6037f73da76d: Status 404 returned error can't find the container with id 93e88cd5806cd45d635fb879e9fdbf90d47b68274ae6a9cb009a6037f73da76d Jul 09 23:27:55.711510 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.711480 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72e31df_6cc8_4ab8_88f7_0863b285e03f.slice/crio-76316d97636c38b23394ffa891f14f7db9e39642d46b8065d4cf5bb5b01a88df WatchSource:0}: Error finding container 76316d97636c38b23394ffa891f14f7db9e39642d46b8065d4cf5bb5b01a88df: Status 404 returned error can't find the container with id 76316d97636c38b23394ffa891f14f7db9e39642d46b8065d4cf5bb5b01a88df Jul 09 23:27:55.717922 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.717865 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2cbebe_d114_4c91_b31f_9d2698990faa.slice/crio-c17e8aa9e54f66787d5205efe2f6b4c1d3fcc5324e4666e1776f36c779581291 WatchSource:0}: Error finding container c17e8aa9e54f66787d5205efe2f6b4c1d3fcc5324e4666e1776f36c779581291: Status 404 returned error can't find the container with id c17e8aa9e54f66787d5205efe2f6b4c1d3fcc5324e4666e1776f36c779581291 Jul 09 23:27:55.721804 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:27:55.721762 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d45b59a_6c66_4708_afb4_0efcbf6ca153.slice/crio-c48357794ff2f728c10e95d2ccf5871e5aeb703fbdf3bb7485e6bdacb416fbb9 WatchSource:0}: Error finding container c48357794ff2f728c10e95d2ccf5871e5aeb703fbdf3bb7485e6bdacb416fbb9: Status 404 returned error can't find the container with id c48357794ff2f728c10e95d2ccf5871e5aeb703fbdf3bb7485e6bdacb416fbb9 Jul 09 23:27:56.162134 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.162041 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:56.162332 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.162246 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:56.162386 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.162337 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs podName:d7f4db34-e093-4e81-9bde-54b01feb897f nodeName:}" failed. No retries permitted until 2026-07-09 23:27:58.162313363 +0000 UTC m=+6.043065321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs") pod "network-metrics-daemon-fz2qx" (UID: "d7f4db34-e093-4e81-9bde-54b01feb897f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:56.262730 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.262683 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:56.262924 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.262867 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:56.262987 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.262936 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret podName:b38ad682-bd50-4725-b3c1-821e12e3e15f nodeName:}" failed. No retries permitted until 2026-07-09 23:27:58.262916002 +0000 UTC m=+6.143667958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret") pod "global-pull-secret-syncer-8gm96" (UID: "b38ad682-bd50-4725-b3c1-821e12e3e15f") : object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:56.363970 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.363926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:56.364180 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.364091 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 23:27:56.364180 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.364107 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 23:27:56.364180 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.364117 2562 projected.go:196] Error preparing data for projected volume kube-api-access-t88d9 for pod openshift-network-diagnostics/network-check-target-mhsqr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:56.364180 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.364163 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9 podName:37fa4fdd-fff0-4421-a01b-b8139858c0d4 nodeName:}" failed. No retries permitted until 2026-07-09 23:27:58.364148944 +0000 UTC m=+6.244900884 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t88d9" (UniqueName: "kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9") pod "network-check-target-mhsqr" (UID: "37fa4fdd-fff0-4421-a01b-b8139858c0d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:56.581415 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.581293 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 23:22:55 +0000 UTC" deadline="2028-03-08 13:36:41.08307716 +0000 UTC" Jul 09 23:27:56.581415 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.581362 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14582h8m44.501720191s" Jul 09 23:27:56.616411 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.616370 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:56.616582 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.616513 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:27:56.616985 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.616962 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:56.617107 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.617086 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:27:56.617183 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.617155 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:56.617271 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:56.617223 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:27:56.648537 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.648483 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerStarted","Data":"c17e8aa9e54f66787d5205efe2f6b4c1d3fcc5324e4666e1776f36c779581291"} Jul 09 23:27:56.660356 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.660251 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ng94m" event={"ID":"a72e31df-6cc8-4ab8-88f7-0863b285e03f","Type":"ContainerStarted","Data":"76316d97636c38b23394ffa891f14f7db9e39642d46b8065d4cf5bb5b01a88df"} Jul 09 23:27:56.681678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.681642 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" event={"ID":"02c11d85-60a2-4b3b-a979-7db7080e13e3","Type":"ContainerStarted","Data":"77ab776b49678cbbd91f2b06adfc790cf5338133a34a990f5d05aeff692b97d7"} Jul 09 23:27:56.690955 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.690354 2562 generic.go:356] "Generic (PLEG): container finished" podID="833514e0c442828ea1fb7e0b8882a68d" containerID="fad7b48b082d47f95ed22aca13187f2b99553f20b662dad8739442f0702f7fc2" exitCode=0 Jul 09 23:27:56.690955 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.690454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" event={"ID":"833514e0c442828ea1fb7e0b8882a68d","Type":"ContainerDied","Data":"fad7b48b082d47f95ed22aca13187f2b99553f20b662dad8739442f0702f7fc2"} Jul 09 23:27:56.702026 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.701996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wljd" event={"ID":"52e676c6-c4db-4322-8eb4-d9c56bd4843a","Type":"ContainerStarted","Data":"da7eaf0aab53076eaa167aef6e5a7c07980c5a7a79fb7ea4abada58050f950d1"} Jul 09 23:27:56.708375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.708040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"d9abd26b657e726b26e289d220a4dd2f76a1c102c6963cfd3f75d102edc431f4"} Jul 09 23:27:56.738713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.738673 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vqmv" event={"ID":"b040f8e8-3927-4005-a787-52003adb5e78","Type":"ContainerStarted","Data":"93e88cd5806cd45d635fb879e9fdbf90d47b68274ae6a9cb009a6037f73da76d"} Jul 09 23:27:56.753542 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.753507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hvswt" event={"ID":"c3f7c514-15d5-4ff6-a32f-de5e43a59120","Type":"ContainerStarted","Data":"8a737298ea255f5f5ede16b575bc02f60f82a3a94dc1f748c484fc76f4284515"} Jul 09 23:27:56.778018 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.774168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8gb8" event={"ID":"6cdbbd61-cd46-4dea-a427-322f8815faca","Type":"ContainerStarted","Data":"3c569acd560960fb1026bfd744cfe25d3be273eb6f46aee2070a0d56e53257be"} Jul 09 23:27:56.788083 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.788018 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" event={"ID":"68d79ca2f41a1fb714c439c75beb926d","Type":"ContainerStarted","Data":"e0d78672252ca91687d9a96e0ee39bdf0133213dcbaf9a2e0b522d3372aed0aa"} Jul 09 23:27:56.801565 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.801478 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-88.ec2.internal" podStartSLOduration=2.801454204 podStartE2EDuration="2.801454204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:27:56.801418274 +0000 UTC m=+4.682170238" watchObservedRunningTime="2026-07-09 23:27:56.801454204 +0000 UTC m=+4.682206172" Jul 09 23:27:56.802512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:56.802480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t59c6" event={"ID":"8d45b59a-6c66-4708-afb4-0efcbf6ca153","Type":"ContainerStarted","Data":"c48357794ff2f728c10e95d2ccf5871e5aeb703fbdf3bb7485e6bdacb416fbb9"} Jul 09 23:27:57.826061 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:57.825992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" event={"ID":"833514e0c442828ea1fb7e0b8882a68d","Type":"ContainerStarted","Data":"4ff2d6e8a2d538ea72e0900b53d5097cd5f73894a5756c9c89356f4901330656"} Jul 09 23:27:58.180169 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:58.180079 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:58.180363 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.180283 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:58.180363 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.180356 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs podName:d7f4db34-e093-4e81-9bde-54b01feb897f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:02.1803348 +0000 UTC m=+10.061086740 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs") pod "network-metrics-daemon-fz2qx" (UID: "d7f4db34-e093-4e81-9bde-54b01feb897f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:27:58.281755 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:58.281145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:58.281755 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.281316 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:58.281755 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.281384 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret podName:b38ad682-bd50-4725-b3c1-821e12e3e15f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:02.281365059 +0000 UTC m=+10.162117012 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret") pod "global-pull-secret-syncer-8gm96" (UID: "b38ad682-bd50-4725-b3c1-821e12e3e15f") : object "kube-system"/"original-pull-secret" not registered Jul 09 23:27:58.382650 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:58.381937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:58.382650 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.382152 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 23:27:58.382650 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.382175 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 23:27:58.382650 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.382208 2562 projected.go:196] Error preparing data for projected volume kube-api-access-t88d9 for pod openshift-network-diagnostics/network-check-target-mhsqr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:58.382650 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.382283 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9 podName:37fa4fdd-fff0-4421-a01b-b8139858c0d4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:02.382268253 +0000 UTC m=+10.263020192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t88d9" (UniqueName: "kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9") pod "network-check-target-mhsqr" (UID: "37fa4fdd-fff0-4421-a01b-b8139858c0d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:27:58.612332 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:58.612294 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:27:58.612517 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.612443 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:27:58.612904 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:58.612879 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:27:58.613021 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.613004 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:27:58.613102 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:27:58.613063 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:27:58.613161 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:27:58.613129 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:00.612586 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:00.612545 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:00.613093 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:00.612707 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:00.613190 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:00.613119 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:00.613386 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:00.613232 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:00.613386 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:00.613305 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:00.613386 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:00.613377 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:02.215172 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:02.215136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:02.215688 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.215293 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:28:02.215688 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.215354 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs podName:d7f4db34-e093-4e81-9bde-54b01feb897f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:10.215337603 +0000 UTC m=+18.096089542 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs") pod "network-metrics-daemon-fz2qx" (UID: "d7f4db34-e093-4e81-9bde-54b01feb897f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:28:02.316476 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:02.316439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:02.316646 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.316575 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 23:28:02.316646 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.316623 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret podName:b38ad682-bd50-4725-b3c1-821e12e3e15f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:10.316609031 +0000 UTC m=+18.197360971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret") pod "global-pull-secret-syncer-8gm96" (UID: "b38ad682-bd50-4725-b3c1-821e12e3e15f") : object "kube-system"/"original-pull-secret" not registered Jul 09 23:28:02.418014 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:02.417966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:02.418200 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.418188 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 23:28:02.418284 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.418211 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 23:28:02.418284 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.418232 2562 projected.go:196] Error preparing data for projected volume kube-api-access-t88d9 for pod openshift-network-diagnostics/network-check-target-mhsqr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:28:02.418414 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.418324 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9 podName:37fa4fdd-fff0-4421-a01b-b8139858c0d4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:10.418296103 +0000 UTC m=+18.299048062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t88d9" (UniqueName: "kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9") pod "network-check-target-mhsqr" (UID: "37fa4fdd-fff0-4421-a01b-b8139858c0d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:28:02.615808 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:02.615776 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:02.615998 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:02.615907 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:02.615998 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.615925 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:02.616132 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.616053 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:02.616200 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:02.616134 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:02.616281 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:02.616260 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:04.615723 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:04.615693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:04.616131 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:04.615818 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:04.616131 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:04.615827 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:04.616131 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:04.615836 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:04.616131 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:04.615954 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:04.616131 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:04.616041 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:06.612218 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:06.612178 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:06.612669 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:06.612224 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:06.612669 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:06.612319 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:06.612669 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:06.612374 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:06.612669 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:06.612510 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:06.612669 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:06.612598 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:08.615187 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:08.615149 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:08.615187 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:08.615189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:08.615619 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:08.615285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:08.615619 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:08.615287 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:08.615619 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:08.615411 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:08.615619 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:08.615480 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:10.282841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:10.282810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:10.283428 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.283003 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:28:10.283428 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.283082 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs podName:d7f4db34-e093-4e81-9bde-54b01feb897f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:26.283061581 +0000 UTC m=+34.163813532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs") pod "network-metrics-daemon-fz2qx" (UID: "d7f4db34-e093-4e81-9bde-54b01feb897f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:28:10.383985 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:10.383944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:10.384146 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.384082 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 23:28:10.384146 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.384133 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret podName:b38ad682-bd50-4725-b3c1-821e12e3e15f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:26.384120613 +0000 UTC m=+34.264872551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret") pod "global-pull-secret-syncer-8gm96" (UID: "b38ad682-bd50-4725-b3c1-821e12e3e15f") : object "kube-system"/"original-pull-secret" not registered Jul 09 23:28:10.485215 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:10.485167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:10.485374 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.485356 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 23:28:10.485429 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.485381 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 23:28:10.485429 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.485391 2562 projected.go:196] Error preparing data for projected volume kube-api-access-t88d9 for pod openshift-network-diagnostics/network-check-target-mhsqr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:28:10.485492 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.485441 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9 podName:37fa4fdd-fff0-4421-a01b-b8139858c0d4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:26.485426594 +0000 UTC m=+34.366178535 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t88d9" (UniqueName: "kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9") pod "network-check-target-mhsqr" (UID: "37fa4fdd-fff0-4421-a01b-b8139858c0d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:28:10.612171 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:10.612086 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:10.612356 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:10.612085 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:10.612356 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.612274 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:10.612356 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:10.612090 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:10.612356 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.612319 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:10.612547 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:10.612401 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:12.612462 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:12.612433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:12.612936 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:12.612540 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:12.612936 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:12.612702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:12.612936 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:12.612792 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:12.612936 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:12.612849 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:12.612936 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:12.612903 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:14.612044 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:14.612008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:14.612534 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:14.612008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:14.612534 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:14.612134 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:14.612534 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:14.612189 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:14.612534 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:14.612008 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:14.612534 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:14.612297 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:15.873669 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:15.871052 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wljd" event={"ID":"52e676c6-c4db-4322-8eb4-d9c56bd4843a","Type":"ContainerStarted","Data":"82377f3ce6192a89552b8cfdd8a4a13406d03982e8ffd9c6d6aa94ce59f81f1b"} Jul 09 23:28:15.877898 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:15.877853 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hvswt" event={"ID":"c3f7c514-15d5-4ff6-a32f-de5e43a59120","Type":"ContainerStarted","Data":"760800a32559c3bf54c4a4b8ac5aea7f193178688b32cebb324c16b3221b0389"} Jul 09 23:28:15.884122 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:15.883324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8gb8" event={"ID":"6cdbbd61-cd46-4dea-a427-322f8815faca","Type":"ContainerStarted","Data":"ce829dfa8dc442b4538bc5e9e6110ef46c25f3153491754f2b0c1f467fdd3f99"} Jul 09 23:28:15.899495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:15.899440 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hvswt" podStartSLOduration=2.8456513230000002 podStartE2EDuration="22.899419172s" totalImagesPullingTime="20.053767849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:15.898769204 +0000 UTC m=+23.779521165" watchObservedRunningTime="2026-07-09 23:28:15.899419172 +0000 UTC m=+23.780171132" Jul 09 23:28:15.899687 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:15.899638 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-88.ec2.internal" podStartSLOduration=21.899626355 podStartE2EDuration="21.899626355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:27:57.842152631 +0000 UTC m=+5.722904597" watchObservedRunningTime="2026-07-09 23:28:15.899626355 +0000 UTC m=+23.780378317" Jul 09 23:28:15.913887 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:15.913841 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g8gb8" podStartSLOduration=7.055522371 podStartE2EDuration="22.913821702s" totalImagesPullingTime="15.858299331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:15.913695662 +0000 UTC m=+23.794447623" watchObservedRunningTime="2026-07-09 23:28:15.913821702 +0000 UTC m=+23.794573663" Jul 09 23:28:16.612618 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.612457 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:16.612841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.612457 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:16.612841 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:16.612718 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:16.612841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.612504 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:16.612841 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:16.612773 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:16.612841 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:16.612839 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:16.888191 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888129 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:28:16.888713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888465 2562 generic.go:356] "Generic (PLEG): container finished" podID="eb3d9b13-8e9b-4501-883a-a8742ae6068a" containerID="cf0b5b70fb0ae63e740c9691bd37fad6733bcaf1d96db9de53b0de32b5a29ec8" exitCode=1 Jul 09 23:28:16.888713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"d1ba3c1b245d0877c7403be68a5da33f92801285e69a9ea5de17e404740720bf"} Jul 09 23:28:16.888713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888593 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"2e5272f033ff635d517ee84dd9aae3b67a5b3dd3082cb617996597525a6797c0"} Jul 09 23:28:16.888713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"b57440c15d22f2f7f35a53cbc0a516115e08f19ba72a93bf76f8e505e2b67af7"} Jul 09 23:28:16.888713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888658 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"2c1dd458bb8e3afa2049eb6a8f05eec7d363980db2e630bca6ebe2124f5097dd"} Jul 09 23:28:16.888713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerDied","Data":"cf0b5b70fb0ae63e740c9691bd37fad6733bcaf1d96db9de53b0de32b5a29ec8"} Jul 09 23:28:16.888957 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.888726 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"2a0d8740b75af4e5f792a9d706b2a08e1b89db3c3ccf49171283773054206ff3"} Jul 09 23:28:16.889794 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.889754 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8vqmv" event={"ID":"b040f8e8-3927-4005-a787-52003adb5e78","Type":"ContainerStarted","Data":"9bbb1fb1e77184627cdc0d144db46e82d87e320e054b75977e9cd20aae41532f"} Jul 09 23:28:16.890990 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.890973 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-t59c6" event={"ID":"8d45b59a-6c66-4708-afb4-0efcbf6ca153","Type":"ContainerStarted","Data":"538041eeb4d932d395489741feacb5e3f3728dd158e247978b626ba28a2ff77a"} Jul 09 23:28:16.892323 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.892304 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef2cbebe-d114-4c91-b31f-9d2698990faa" containerID="c8e921c0e882c884512b1fada5adb7ecf35fd1fd615482c3058acffc3aa4d245" exitCode=0 Jul 09 23:28:16.892409 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.892360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerDied","Data":"c8e921c0e882c884512b1fada5adb7ecf35fd1fd615482c3058acffc3aa4d245"} Jul 09 23:28:16.893707 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.893685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" event={"ID":"02c11d85-60a2-4b3b-a979-7db7080e13e3","Type":"ContainerStarted","Data":"3e9dbe5bf58c953409cc9ac701d9ae1fa5c92513ce9a81c8bfafea992aec66e9"} Jul 09 23:28:16.918945 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.918893 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8vqmv" podStartSLOduration=8.095666644 podStartE2EDuration="23.918878445s" totalImagesPullingTime="15.823211801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:16.906151139 +0000 UTC m=+24.786903099" watchObservedRunningTime="2026-07-09 23:28:16.918878445 +0000 UTC m=+24.799630473" Jul 09 23:28:16.919087 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.919064 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-t59c6" podStartSLOduration=3.943937298 podStartE2EDuration="23.919058101s" totalImagesPullingTime="19.975120803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:16.918686185 +0000 UTC m=+24.799438202" watchObservedRunningTime="2026-07-09 23:28:16.919058101 +0000 UTC m=+24.799810061" Jul 09 23:28:16.953005 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:16.952951 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wljd" podStartSLOduration=3.949588218 podStartE2EDuration="23.952932617s" totalImagesPullingTime="20.003344399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:16.952470259 +0000 UTC m=+24.833222216" watchObservedRunningTime="2026-07-09 23:28:16.952932617 +0000 UTC m=+24.833684579" Jul 09 23:28:17.214722 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:17.214696 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 23:28:17.609051 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:17.608944 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T23:28:17.21471628Z","UUID":"e4a080fa-cdea-49bf-8ca2-7d7ed30573a1","Handler":null,"Name":"","Endpoint":""} Jul 09 23:28:17.610505 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:17.610478 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 23:28:17.610645 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:17.610522 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 23:28:17.897358 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:17.897219 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ng94m" event={"ID":"a72e31df-6cc8-4ab8-88f7-0863b285e03f","Type":"ContainerStarted","Data":"7a36f0ceb5f6eed7a6eda23e4ba4d0efa7ad4b675706fdb4dcddabb49aa5bf95"} Jul 09 23:28:17.899764 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:17.899697 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" event={"ID":"02c11d85-60a2-4b3b-a979-7db7080e13e3","Type":"ContainerStarted","Data":"c5c950c4afeeab68a7bac5affce58984b4d687d4d0d158eaaec6c10023f0228d"} Jul 09 23:28:17.911972 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:17.911924 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ng94m" podStartSLOduration=4.930944145 podStartE2EDuration="24.911905876s" totalImagesPullingTime="19.980961731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:17.911117028 +0000 UTC m=+25.791868991" watchObservedRunningTime="2026-07-09 23:28:17.911905876 +0000 UTC m=+25.792657835" Jul 09 23:28:18.031713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.031674 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:28:18.032361 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.032340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:28:18.612604 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.612418 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:18.612795 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:18.612710 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:18.612795 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.612469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:18.612922 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:18.612803 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:18.612922 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.612441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:18.612922 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:18.612875 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:18.904383 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.904296 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" event={"ID":"02c11d85-60a2-4b3b-a979-7db7080e13e3","Type":"ContainerStarted","Data":"7fd8f98e1130ae3b4e6a54f76b70a64dc6d0c7f9ddaba671b6ba2f7d955e81f1"} Jul 09 23:28:18.908255 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.908085 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:28:18.908483 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.908458 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"178bb1f0d04957048149e821d4158ff4421f5939b178bc1fc8e708dccd3d0272"} Jul 09 23:28:18.908904 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.908878 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:28:18.909742 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.909727 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8vqmv" Jul 09 23:28:18.921137 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:18.921087 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qw9" podStartSLOduration=3.730959894 podStartE2EDuration="25.921069781s" totalImagesPullingTime="22.190109887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:18.919937899 +0000 UTC m=+26.800689859" watchObservedRunningTime="2026-07-09 23:28:18.921069781 +0000 UTC m=+26.801821743" Jul 09 23:28:20.615313 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:20.615281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:20.615916 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:20.615390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:20.615916 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:20.615400 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:20.615916 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:20.615431 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:20.615916 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:20.615532 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:20.615916 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:20.615597 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:21.918843 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:21.918683 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:28:21.919430 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:21.919195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"d17baa19dc691b4721e657e1019f0490a9e1c87219e440a9349f98461c4fb88d"} Jul 09 23:28:21.919534 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:21.919516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:28:21.919642 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:21.919629 2562 scope.go:117] "RemoveContainer" containerID="cf0b5b70fb0ae63e740c9691bd37fad6733bcaf1d96db9de53b0de32b5a29ec8" Jul 09 23:28:21.921146 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:21.921120 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef2cbebe-d114-4c91-b31f-9d2698990faa" containerID="e4e34937dd848574214ced4bdab7d0121690022bc955341dde4b52a0a2fe2522" exitCode=0 Jul 09 23:28:21.921271 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:21.921175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerDied","Data":"e4e34937dd848574214ced4bdab7d0121690022bc955341dde4b52a0a2fe2522"} Jul 09 23:28:21.936300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:21.936275 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:28:22.631877 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.631846 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:22.632161 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:22.631975 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:22.632161 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.632146 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:22.632309 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:22.632251 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:22.632365 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.632314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:22.632414 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:22.632395 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:22.927440 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.927367 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:28:22.927870 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.927724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" event={"ID":"eb3d9b13-8e9b-4501-883a-a8742ae6068a","Type":"ContainerStarted","Data":"f215aa5e5de216215ee4af051bd39742835212ceb04c7c627ff72c05dd1d6be9"} Jul 09 23:28:22.927870 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.927839 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 23:28:22.928113 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.928096 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:28:22.944188 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.944167 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:28:22.958039 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:22.958000 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" podStartSLOduration=9.873481018 podStartE2EDuration="29.957985917s" totalImagesPullingTime="20.084504899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:22.957822635 +0000 UTC m=+30.838574595" watchObservedRunningTime="2026-07-09 23:28:22.957985917 +0000 UTC m=+30.838737877" Jul 09 23:28:23.287337 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.287111 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gm96"] Jul 09 23:28:23.287517 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.287416 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:23.287517 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:23.287500 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:23.290373 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.290344 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhsqr"] Jul 09 23:28:23.290519 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.290454 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:23.290578 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:23.290556 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:23.290967 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.290939 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fz2qx"] Jul 09 23:28:23.291087 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.291042 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:23.291154 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:23.291134 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:23.931778 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.931742 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef2cbebe-d114-4c91-b31f-9d2698990faa" containerID="1d63dfa8b63cf8b8450b29379a23497440658d4a6288d5218552712d0da339c9" exitCode=0 Jul 09 23:28:23.932153 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.931816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerDied","Data":"1d63dfa8b63cf8b8450b29379a23497440658d4a6288d5218552712d0da339c9"} Jul 09 23:28:23.932153 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:23.931964 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 23:28:24.487597 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:24.487556 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:28:24.588373 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:24.588338 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:24.615614 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:24.615590 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:24.615614 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:24.615599 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:24.615614 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:24.615606 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:24.615847 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:24.615684 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:24.615847 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:24.615755 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:24.615847 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:24.615810 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:24.676545 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:24.676517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:24.777039 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:24.777012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:25.939029 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:25.938990 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef2cbebe-d114-4c91-b31f-9d2698990faa" containerID="9d106258de14f42218385d29c1725884e95b4e4fbb3dbda0df6148f9284fe0b0" exitCode=0 Jul 09 23:28:25.939958 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:25.939073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerDied","Data":"9d106258de14f42218385d29c1725884e95b4e4fbb3dbda0df6148f9284fe0b0"} Jul 09 23:28:26.257547 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.257515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:26.299643 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.299619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:26.299782 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.299732 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:28:26.299817 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.299783 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs podName:d7f4db34-e093-4e81-9bde-54b01feb897f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:58.299769054 +0000 UTC m=+66.180520992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs") pod "network-metrics-daemon-fz2qx" (UID: "d7f4db34-e093-4e81-9bde-54b01feb897f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 23:28:26.333185 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.333145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:26.395748 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.395714 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:26.400781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.400755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:26.400921 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.400901 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 23:28:26.401005 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.400993 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret podName:b38ad682-bd50-4725-b3c1-821e12e3e15f nodeName:}" failed. No retries permitted until 2026-07-09 23:28:58.400972737 +0000 UTC m=+66.281724688 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret") pod "global-pull-secret-syncer-8gm96" (UID: "b38ad682-bd50-4725-b3c1-821e12e3e15f") : object "kube-system"/"original-pull-secret" not registered Jul 09 23:28:26.501125 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.501071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:26.501317 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.501296 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 23:28:26.501372 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.501329 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 23:28:26.501372 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.501345 2562 projected.go:196] Error preparing data for projected volume kube-api-access-t88d9 for pod openshift-network-diagnostics/network-check-target-mhsqr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:28:26.501454 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.501411 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9 podName:37fa4fdd-fff0-4421-a01b-b8139858c0d4 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:58.501392999 +0000 UTC m=+66.382144950 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t88d9" (UniqueName: "kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9") pod "network-check-target-mhsqr" (UID: "37fa4fdd-fff0-4421-a01b-b8139858c0d4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 23:28:26.612227 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.612130 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:26.612227 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.612130 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:26.612441 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.612272 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mhsqr" podUID="37fa4fdd-fff0-4421-a01b-b8139858c0d4" Jul 09 23:28:26.612441 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.612308 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:26.613131 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.613096 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fz2qx" podUID="d7f4db34-e093-4e81-9bde-54b01feb897f" Jul 09 23:28:26.613279 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:26.613257 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gm96" podUID="b38ad682-bd50-4725-b3c1-821e12e3e15f" Jul 09 23:28:26.991755 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:26.991665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:27.389929 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:27.389895 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:27.489081 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:27.489047 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:28.457346 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.457267 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-88.ec2.internal" event="NodeReady" Jul 09 23:28:28.457704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.457418 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 23:28:28.488501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.488470 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59b874555-jdqsx"] Jul 09 23:28:28.492577 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.492549 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.494994 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.494962 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 23:28:28.495107 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.495088 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 23:28:28.495189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.495110 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 23:28:28.495189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.495088 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gjrpj\"" Jul 09 23:28:28.497622 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.497599 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7zb56"] Jul 09 23:28:28.500132 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.500108 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 23:28:28.501029 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.501008 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nsrv8"] Jul 09 23:28:28.501122 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.501087 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:28.503319 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.503298 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 23:28:28.503460 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.503391 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4bz2v\"" Jul 09 23:28:28.504426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.504221 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 23:28:28.504559 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.504288 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 23:28:28.505568 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.505550 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59b874555-jdqsx"] Jul 09 23:28:28.505672 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.505654 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.508174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.507925 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8548g\"" Jul 09 23:28:28.508174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.507937 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 23:28:28.508174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.508029 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 23:28:28.510698 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.510631 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7zb56"] Jul 09 23:28:28.513640 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.513616 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nsrv8"] Jul 09 23:28:28.612079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.612046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:28.612272 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.612056 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:28.612272 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.612244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:28.614598 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.614572 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 23:28:28.614720 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.614643 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r9l9m\"" Jul 09 23:28:28.615029 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.615009 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8rpsb\"" Jul 09 23:28:28.615132 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.615113 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 23:28:28.615214 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.615018 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 23:28:28.615273 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.615228 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 23:28:28.617088 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617066 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67a109cb-dc48-4065-a414-64c89161ff14-ca-trust-extracted\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.617193 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-trusted-ca\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.617193 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617129 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-bound-sa-token\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.617193 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-registry-certificates\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.617193 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617184 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x684j\" (UniqueName: \"kubernetes.io/projected/8ab4bf87-1ebf-4b1f-b483-65ecd08136fb-kube-api-access-x684j\") pod \"ingress-canary-7zb56\" (UID: \"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb\") " pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:28.617410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.617410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-image-registry-private-configuration\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.617410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxvz\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-kube-api-access-bmxvz\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.617410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617347 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ab4bf87-1ebf-4b1f-b483-65ecd08136fb-cert\") pod \"ingress-canary-7zb56\" (UID: \"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb\") " pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:28.617410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617370 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-config-volume\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.617410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-tmp-dir\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.617678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.617678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ch6\" (UniqueName: \"kubernetes.io/projected/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-kube-api-access-q2ch6\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.617678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.617536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-installation-pull-secrets\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.621726 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.621703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:28.718467 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718368 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ab4bf87-1ebf-4b1f-b483-65ecd08136fb-cert\") pod \"ingress-canary-7zb56\" (UID: \"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb\") " pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:28.718467 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-config-volume\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-tmp-dir\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ch6\" (UniqueName: \"kubernetes.io/projected/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-kube-api-access-q2ch6\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718561 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-installation-pull-secrets\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67a109cb-dc48-4065-a414-64c89161ff14-ca-trust-extracted\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:28.718644 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-trusted-ca\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.718689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-bound-sa-token\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.719058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-registry-certificates\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.719058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718736 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x684j\" (UniqueName: \"kubernetes.io/projected/8ab4bf87-1ebf-4b1f-b483-65ecd08136fb-kube-api-access-x684j\") pod \"ingress-canary-7zb56\" (UID: \"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb\") " pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:28.719058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.719058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-image-registry-private-configuration\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.719058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxvz\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-kube-api-access-bmxvz\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.719058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.718846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-tmp-dir\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.719058 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:28.718659 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b874555-jdqsx: secret "image-registry-tls" not found Jul 09 23:28:28.719414 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:28.719101 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls podName:67a109cb-dc48-4065-a414-64c89161ff14 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:29.219082143 +0000 UTC m=+37.099834088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls") pod "image-registry-59b874555-jdqsx" (UID: "67a109cb-dc48-4065-a414-64c89161ff14") : secret "image-registry-tls" not found Jul 09 23:28:28.719414 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.719102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-config-volume\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.719414 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:28.719227 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 23:28:28.719414 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.719237 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67a109cb-dc48-4065-a414-64c89161ff14-ca-trust-extracted\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.719414 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:28.719279 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls podName:a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c nodeName:}" failed. No retries permitted until 2026-07-09 23:28:29.219260945 +0000 UTC m=+37.100012885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls") pod "dns-default-nsrv8" (UID: "a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c") : secret "dns-default-metrics-tls" not found Jul 09 23:28:28.719708 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.719682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-registry-certificates\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.719908 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.719854 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-trusted-ca\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.721381 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.721238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:28.723551 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.723528 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-image-registry-private-configuration\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.723645 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.723602 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ab4bf87-1ebf-4b1f-b483-65ecd08136fb-cert\") pod \"ingress-canary-7zb56\" (UID: \"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb\") " pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:28.723942 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.723921 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-installation-pull-secrets\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.728032 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.728006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-bound-sa-token\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.728128 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.728092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ch6\" (UniqueName: \"kubernetes.io/projected/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-kube-api-access-q2ch6\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:28.728238 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.728194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxvz\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-kube-api-access-bmxvz\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:28.729196 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.729177 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x684j\" (UniqueName: \"kubernetes.io/projected/8ab4bf87-1ebf-4b1f-b483-65ecd08136fb-kube-api-access-x684j\") pod \"ingress-canary-7zb56\" (UID: \"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb\") " pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:28.815184 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:28.815151 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7zb56" Jul 09 23:28:29.223060 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:29.223022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:29.223254 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:29.223118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:29.223254 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:29.223234 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 23:28:29.223254 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:29.223245 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 23:28:29.223254 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:29.223254 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b874555-jdqsx: secret "image-registry-tls" not found Jul 09 23:28:29.223409 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:29.223316 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls podName:67a109cb-dc48-4065-a414-64c89161ff14 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:30.22329606 +0000 UTC m=+38.104048015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls") pod "image-registry-59b874555-jdqsx" (UID: "67a109cb-dc48-4065-a414-64c89161ff14") : secret "image-registry-tls" not found Jul 09 23:28:29.223409 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:29.223332 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls podName:a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c nodeName:}" failed. No retries permitted until 2026-07-09 23:28:30.223324938 +0000 UTC m=+38.104076877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls") pod "dns-default-nsrv8" (UID: "a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c") : secret "dns-default-metrics-tls" not found Jul 09 23:28:29.621899 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:29.621869 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:29.986789 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:29.986759 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7zb56"] Jul 09 23:28:29.990598 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:28:29.990570 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab4bf87_1ebf_4b1f_b483_65ecd08136fb.slice/crio-c12b4b56d48955785672523ea81c328660819310ca9b3a9cffd1b41bab81174c WatchSource:0}: Error finding container c12b4b56d48955785672523ea81c328660819310ca9b3a9cffd1b41bab81174c: Status 404 returned error can't find the container with id c12b4b56d48955785672523ea81c328660819310ca9b3a9cffd1b41bab81174c Jul 09 23:28:30.231288 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:30.231146 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:30.231288 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:30.231268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:30.231489 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:30.231314 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 23:28:30.231489 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:30.231331 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b874555-jdqsx: secret "image-registry-tls" not found Jul 09 23:28:30.231489 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:30.231369 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 23:28:30.231489 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:30.231410 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls podName:67a109cb-dc48-4065-a414-64c89161ff14 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:32.231368287 +0000 UTC m=+40.112120224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls") pod "image-registry-59b874555-jdqsx" (UID: "67a109cb-dc48-4065-a414-64c89161ff14") : secret "image-registry-tls" not found Jul 09 23:28:30.231489 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:30.231424 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls podName:a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c nodeName:}" failed. No retries permitted until 2026-07-09 23:28:32.231418336 +0000 UTC m=+40.112170274 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls") pod "dns-default-nsrv8" (UID: "a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c") : secret "dns-default-metrics-tls" not found Jul 09 23:28:30.954570 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:30.954526 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef2cbebe-d114-4c91-b31f-9d2698990faa" containerID="ce46a3e6790dc57370e66a1df4af17d98d039aa2435882da8cff73a8662b4f32" exitCode=0 Jul 09 23:28:30.955030 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:30.954609 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerDied","Data":"ce46a3e6790dc57370e66a1df4af17d98d039aa2435882da8cff73a8662b4f32"} Jul 09 23:28:30.956214 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:30.956175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7zb56" event={"ID":"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb","Type":"ContainerStarted","Data":"c12b4b56d48955785672523ea81c328660819310ca9b3a9cffd1b41bab81174c"} Jul 09 23:28:31.002108 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:31.001156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:31.057163 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:31.057107 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:31.161149 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:31.160963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:31.962158 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:31.962121 2562 generic.go:356] "Generic (PLEG): container finished" podID="ef2cbebe-d114-4c91-b31f-9d2698990faa" containerID="866bc6e21cbb277a3b54ece7b4c73bce30a592df95f72713795f2da60fb0ee0d" exitCode=0 Jul 09 23:28:31.962637 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:31.962184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerDied","Data":"866bc6e21cbb277a3b54ece7b4c73bce30a592df95f72713795f2da60fb0ee0d"} Jul 09 23:28:32.247599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:32.247559 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:32.247707 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:32.247657 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:32.247771 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:32.247709 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 23:28:32.247771 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:32.247728 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b874555-jdqsx: secret "image-registry-tls" not found Jul 09 23:28:32.247844 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:32.247783 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls podName:67a109cb-dc48-4065-a414-64c89161ff14 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:36.247768773 +0000 UTC m=+44.128520718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls") pod "image-registry-59b874555-jdqsx" (UID: "67a109cb-dc48-4065-a414-64c89161ff14") : secret "image-registry-tls" not found Jul 09 23:28:32.247844 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:32.247783 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 23:28:32.247844 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:32.247839 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls podName:a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c nodeName:}" failed. No retries permitted until 2026-07-09 23:28:36.247822472 +0000 UTC m=+44.128574424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls") pod "dns-default-nsrv8" (UID: "a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c") : secret "dns-default-metrics-tls" not found Jul 09 23:28:32.965928 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:32.965893 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7zb56" event={"ID":"8ab4bf87-1ebf-4b1f-b483-65ecd08136fb","Type":"ContainerStarted","Data":"164e1ec39aefec8984ceceeabb01e33721cdea40e16ae3f81172bef2356414ed"} Jul 09 23:28:32.969146 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:32.969120 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2xx52" event={"ID":"ef2cbebe-d114-4c91-b31f-9d2698990faa","Type":"ContainerStarted","Data":"785ed7ae417d38fd2504825418d8b5c0942199883d4d1fe4d1bb1458d32972e9"} Jul 09 23:28:32.980848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:32.980800 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7zb56" podStartSLOduration=2.748474855 podStartE2EDuration="4.980784324s" totalImagesPullingTime="2.232309469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:32.980282714 +0000 UTC m=+40.861034675" watchObservedRunningTime="2026-07-09 23:28:32.980784324 +0000 UTC m=+40.861536317" Jul 09 23:28:33.003980 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:33.003928 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2xx52" podStartSLOduration=9.506183206 podStartE2EDuration="40.003913122s" totalImagesPullingTime="30.497729916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:33.002467212 +0000 UTC m=+40.883219174" watchObservedRunningTime="2026-07-09 23:28:33.003913122 +0000 UTC m=+40.884665083" Jul 09 23:28:33.122008 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:33.121913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:33.173822 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:33.173790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:34.002321 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:34.002285 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:35.952556 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:35.952519 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:35.979407 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:35.979375 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf"] Jul 09 23:28:36.015508 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.015467 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf"] Jul 09 23:28:36.015644 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.015607 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" Jul 09 23:28:36.018035 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.018008 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kjmjg\"" Jul 09 23:28:36.018181 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.018008 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 23:28:36.019102 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.019081 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 23:28:36.035345 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.035317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:36.087700 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.087666 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gwjxn"] Jul 09 23:28:36.107215 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.107168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:36.107918 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.107892 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwjxn"] Jul 09 23:28:36.108037 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.108018 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.110913 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.110867 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 23:28:36.110913 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.110881 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 23:28:36.111125 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.110939 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qktl8\"" Jul 09 23:28:36.111125 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.110869 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 23:28:36.111298 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.111287 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 23:28:36.182981 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.182941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8dk\" (UniqueName: \"kubernetes.io/projected/d6b2446b-4b1a-4954-97c7-531a809ec111-kube-api-access-nn8dk\") pod \"migrator-7f59b54f6c-597qf\" (UID: \"d6b2446b-4b1a-4954-97c7-531a809ec111\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" Jul 09 23:28:36.197608 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.197582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:36.283820 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.283782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5rh\" (UniqueName: \"kubernetes.io/projected/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-api-access-rc5rh\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.283992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.283837 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:36.283992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.283862 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.283992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.283902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.283992 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.283937 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 23:28:36.283992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.283950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:36.283992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.283982 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8dk\" (UniqueName: \"kubernetes.io/projected/d6b2446b-4b1a-4954-97c7-531a809ec111-kube-api-access-nn8dk\") pod \"migrator-7f59b54f6c-597qf\" (UID: \"d6b2446b-4b1a-4954-97c7-531a809ec111\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" Jul 09 23:28:36.284290 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.283996 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls podName:a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c nodeName:}" failed. No retries permitted until 2026-07-09 23:28:44.283981394 +0000 UTC m=+52.164733331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls") pod "dns-default-nsrv8" (UID: "a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c") : secret "dns-default-metrics-tls" not found Jul 09 23:28:36.284290 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.284038 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a383ff5-7316-487d-80c8-93f4ca9c6fde-crio-socket\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.284290 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.284067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a383ff5-7316-487d-80c8-93f4ca9c6fde-data-volume\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.284290 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.284092 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 23:28:36.284290 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.284114 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b874555-jdqsx: secret "image-registry-tls" not found Jul 09 23:28:36.284290 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.284176 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls podName:67a109cb-dc48-4065-a414-64c89161ff14 nodeName:}" failed. No retries permitted until 2026-07-09 23:28:44.284159906 +0000 UTC m=+52.164911853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls") pod "image-registry-59b874555-jdqsx" (UID: "67a109cb-dc48-4065-a414-64c89161ff14") : secret "image-registry-tls" not found Jul 09 23:28:36.296412 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.296390 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8dk\" (UniqueName: \"kubernetes.io/projected/d6b2446b-4b1a-4954-97c7-531a809ec111-kube-api-access-nn8dk\") pod \"migrator-7f59b54f6c-597qf\" (UID: \"d6b2446b-4b1a-4954-97c7-531a809ec111\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" Jul 09 23:28:36.324568 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.324540 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" Jul 09 23:28:36.384470 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.384437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a383ff5-7316-487d-80c8-93f4ca9c6fde-crio-socket\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.384470 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.384474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a383ff5-7316-487d-80c8-93f4ca9c6fde-data-volume\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.384692 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.384500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5rh\" (UniqueName: \"kubernetes.io/projected/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-api-access-rc5rh\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.384692 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.384630 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.384692 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.384676 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a383ff5-7316-487d-80c8-93f4ca9c6fde-crio-socket\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.384837 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.384706 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.384837 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.384746 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 23:28:36.384837 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.384804 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 23:28:36.384837 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.384821 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a383ff5-7316-487d-80c8-93f4ca9c6fde-data-volume\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.384981 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.384811 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:36.88479517 +0000 UTC m=+44.765547107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : secret "insights-runtime-extractor-tls" not found Jul 09 23:28:36.384981 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.384878 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:36.884859781 +0000 UTC m=+44.765611744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : configmap "kube-rbac-proxy" not found Jul 09 23:28:36.393574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.393546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5rh\" (UniqueName: \"kubernetes.io/projected/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-api-access-rc5rh\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.479793 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.479763 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf"] Jul 09 23:28:36.495613 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:28:36.495584 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b2446b_4b1a_4954_97c7_531a809ec111.slice/crio-9994dcf50046674c03010b19385b834a8d005b87f689b66555fca2ad7ae285f9 WatchSource:0}: Error finding container 9994dcf50046674c03010b19385b834a8d005b87f689b66555fca2ad7ae285f9: Status 404 returned error can't find the container with id 9994dcf50046674c03010b19385b834a8d005b87f689b66555fca2ad7ae285f9 Jul 09 23:28:36.888637 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.888600 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.888817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.888646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:36.888817 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.888761 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 23:28:36.888917 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.888825 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:37.888811092 +0000 UTC m=+45.769563030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : secret "insights-runtime-extractor-tls" not found Jul 09 23:28:36.888917 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.888761 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 23:28:36.888917 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:36.888886 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:37.888874987 +0000 UTC m=+45.769626925 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : configmap "kube-rbac-proxy" not found Jul 09 23:28:36.979331 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:36.979299 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" event={"ID":"d6b2446b-4b1a-4954-97c7-531a809ec111","Type":"ContainerStarted","Data":"9994dcf50046674c03010b19385b834a8d005b87f689b66555fca2ad7ae285f9"} Jul 09 23:28:37.026172 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.026114 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:37.457485 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.457454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:37.558758 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.558726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:37.731655 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.731580 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:37.829411 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.829375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:37.898547 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.898502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:37.898703 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.898556 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:37.898703 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:37.898650 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 23:28:37.898703 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:37.898690 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 23:28:37.898797 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:37.898717 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:39.89870187 +0000 UTC m=+47.779453808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : secret "insights-runtime-extractor-tls" not found Jul 09 23:28:37.898797 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:37.898738 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:39.898725386 +0000 UTC m=+47.779477324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : configmap "kube-rbac-proxy" not found Jul 09 23:28:37.953297 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:37.953264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:38.223115 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:38.223093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:38.323971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:38.323887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:38.711956 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:38.711878 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t59c6_8d45b59a-6c66-4708-afb4-0efcbf6ca153/dns-node-resolver/0.log" Jul 09 23:28:38.985686 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:38.985599 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" event={"ID":"d6b2446b-4b1a-4954-97c7-531a809ec111","Type":"ContainerStarted","Data":"29a6d6e82d7e0e5db3ed00617b0bf9cd3cff5a7f40a1705dcd0f4e7f0c3fad68"} Jul 09 23:28:38.985686 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:38.985654 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" event={"ID":"d6b2446b-4b1a-4954-97c7-531a809ec111","Type":"ContainerStarted","Data":"c4d7b02de920ff1e4d76860218d337aa6417c3042f5e6eff10736c1cffc9bc5a"} Jul 09 23:28:39.000746 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.000696 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-597qf" podStartSLOduration=2.431304985 podStartE2EDuration="4.000681141s" totalImagesPullingTime="1.569376156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:38.999969956 +0000 UTC m=+46.880721915" watchObservedRunningTime="2026-07-09 23:28:39.000681141 +0000 UTC m=+46.881433079" Jul 09 23:28:39.063773 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.063740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:39.164498 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.164465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:39.380650 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.380609 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:39.435971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.435931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:39.712011 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.711936 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g8gb8_6cdbbd61-cd46-4dea-a427-322f8815faca/node-ca/0.log" Jul 09 23:28:39.914257 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.914197 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:39.914257 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:39.914257 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:39.914484 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:39.914351 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 23:28:39.914484 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:39.914356 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 23:28:39.914484 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:39.914406 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:43.914392335 +0000 UTC m=+51.795144273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : configmap "kube-rbac-proxy" not found Jul 09 23:28:39.914484 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:39.914419 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:43.914412919 +0000 UTC m=+51.795164857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : secret "insights-runtime-extractor-tls" not found Jul 09 23:28:40.112659 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:40.112626 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7zb56_8ab4bf87-1ebf-4b1f-b483-65ecd08136fb/serve-healthcheck-canary/0.log" Jul 09 23:28:40.296399 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:40.296366 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:40.512000 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:40.511969 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-597qf_d6b2446b-4b1a-4954-97c7-531a809ec111/migrator/0.log" Jul 09 23:28:40.711950 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:40.711923 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-597qf_d6b2446b-4b1a-4954-97c7-531a809ec111/graceful-termination/0.log" Jul 09 23:28:41.920781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:41.920749 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:42.832057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:42.832025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:43.285837 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.285803 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:43.469520 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.469492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:43.574091 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.574010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:43.670796 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.670741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:43.722437 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.722410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:43.945949 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.945866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:43.945949 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.945906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:43.946122 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:43.945974 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 23:28:43.946122 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:43.946014 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 23:28:43.946122 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:43.946025 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:51.946010845 +0000 UTC m=+59.826762783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : configmap "kube-rbac-proxy" not found Jul 09 23:28:43.946122 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:43.946105 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:28:51.946087453 +0000 UTC m=+59.826839391 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : secret "insights-runtime-extractor-tls" not found Jul 09 23:28:43.981367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:43.981333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:44.349045 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:44.349014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:28:44.349524 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:44.349085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") pod \"image-registry-59b874555-jdqsx\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:44.349524 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:44.349158 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 23:28:44.349524 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:44.349185 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 23:28:44.349524 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:44.349245 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59b874555-jdqsx: secret "image-registry-tls" not found Jul 09 23:28:44.349524 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:44.349275 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls podName:a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c nodeName:}" failed. No retries permitted until 2026-07-09 23:29:00.349253022 +0000 UTC m=+68.230004973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls") pod "dns-default-nsrv8" (UID: "a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c") : secret "dns-default-metrics-tls" not found Jul 09 23:28:44.349524 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:44.349296 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls podName:67a109cb-dc48-4065-a414-64c89161ff14 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:00.349283834 +0000 UTC m=+68.230035772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls") pod "image-registry-59b874555-jdqsx" (UID: "67a109cb-dc48-4065-a414-64c89161ff14") : secret "image-registry-tls" not found Jul 09 23:28:44.983362 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:44.983330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:45.386277 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:45.386227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:45.578690 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:45.578658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:45.807117 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:45.807086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:46.195265 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:46.195168 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:46.241949 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:46.241912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:46.584632 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:46.584600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:46.973693 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:46.973611 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:47.025905 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:47.025872 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:47.551434 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:47.551396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:47.659410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:47.659382 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:47.784519 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:47.784490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:47.885464 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:47.885386 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:47.986481 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:47.986422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:48.184834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:48.184716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:48.285756 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:48.285726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:48.385229 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:48.385188 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:48.584514 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:48.584484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:48.685926 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:48.685864 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:48.786283 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:48.786190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.029212 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.029172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.129745 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.129713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.230129 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.230096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.279597 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.279521 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.452003 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.451965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.607609 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.607533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.706981 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.706948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:49.981438 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:49.981351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:50.180304 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:50.180274 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:50.280948 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:50.280921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:50.381044 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:50.381018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:50.580566 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:50.580491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:50.783363 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:50.783334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:51.582474 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:51.582431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:52.013174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:52.013145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:52.013567 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:52.013277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:52.013567 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:52.013377 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 23:28:52.013567 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:52.013465 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls podName:0a383ff5-7316-487d-80c8-93f4ca9c6fde nodeName:}" failed. No retries permitted until 2026-07-09 23:29:08.013427214 +0000 UTC m=+75.894179154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwjxn" (UID: "0a383ff5-7316-487d-80c8-93f4ca9c6fde") : secret "insights-runtime-extractor-tls" not found Jul 09 23:28:52.013773 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:52.013738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a383ff5-7316-487d-80c8-93f4ca9c6fde-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:28:52.180701 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:52.180670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:52.382624 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:52.382542 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:52.780071 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:52.780041 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:52.980923 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:52.980847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:53.379855 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:53.379821 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:53.987253 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:53.987215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:54.181985 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:54.181952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:54.380450 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:54.380418 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:55.950573 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:55.950541 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jdkzc" Jul 09 23:28:56.060482 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:56.060432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:56.113947 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:56.113913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:58.213905 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.213876 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95"] Jul 09 23:28:58.216535 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.216514 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b874555-jdqsx"] Jul 09 23:28:58.216664 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.216647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.216717 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:28:58.216695 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-59b874555-jdqsx" podUID="67a109cb-dc48-4065-a414-64c89161ff14" Jul 09 23:28:58.219146 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.219122 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 23:28:58.219146 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.219141 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 09 23:28:58.219325 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.219127 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 23:28:58.219842 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.219828 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 23:28:58.226916 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.226890 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95"] Jul 09 23:28:58.261380 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.261342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3848a6a-3649-4d85-9694-c66d96c7119a-tmp\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.261380 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.261376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f3848a6a-3649-4d85-9694-c66d96c7119a-klusterlet-config\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.261566 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.261472 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x97w5\" (UniqueName: \"kubernetes.io/projected/f3848a6a-3649-4d85-9694-c66d96c7119a-kube-api-access-x97w5\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.309525 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.309489 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-547f948bc6-nmfj5"] Jul 09 23:28:58.311195 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.311172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:58.311409 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.311393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.325547 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.325524 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547f948bc6-nmfj5"] Jul 09 23:28:58.362402 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362379 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7wj\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-kube-api-access-wm7wj\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.362527 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362443 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb779325-7f6b-4c2b-bb97-5db2d4883954-image-registry-private-configuration\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.362527 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3848a6a-3649-4d85-9694-c66d96c7119a-tmp\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.362527 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb779325-7f6b-4c2b-bb97-5db2d4883954-registry-certificates\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.362642 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f3848a6a-3649-4d85-9694-c66d96c7119a-klusterlet-config\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.362719 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362700 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x97w5\" (UniqueName: \"kubernetes.io/projected/f3848a6a-3649-4d85-9694-c66d96c7119a-kube-api-access-x97w5\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.362753 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb779325-7f6b-4c2b-bb97-5db2d4883954-trusted-ca\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.362800 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:58.362800 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362774 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-bound-sa-token\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.362800 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362795 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-registry-tls\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.362930 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb779325-7f6b-4c2b-bb97-5db2d4883954-ca-trust-extracted\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.362930 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3848a6a-3649-4d85-9694-c66d96c7119a-tmp\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.362930 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.362921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb779325-7f6b-4c2b-bb97-5db2d4883954-installation-pull-secrets\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.365063 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.365048 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 23:28:58.365307 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.365289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f3848a6a-3649-4d85-9694-c66d96c7119a-klusterlet-config\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.373506 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.373477 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x97w5\" (UniqueName: \"kubernetes.io/projected/f3848a6a-3649-4d85-9694-c66d96c7119a-kube-api-access-x97w5\") pod \"klusterlet-addon-workmgr-58bfc74fd-4gr95\" (UID: \"f3848a6a-3649-4d85-9694-c66d96c7119a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.374941 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.374923 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7f4db34-e093-4e81-9bde-54b01feb897f-metrics-certs\") pod \"network-metrics-daemon-fz2qx\" (UID: \"d7f4db34-e093-4e81-9bde-54b01feb897f\") " pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:58.407967 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.406031 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:58.463702 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.463668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:58.463812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.463710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb779325-7f6b-4c2b-bb97-5db2d4883954-image-registry-private-configuration\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.463868 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.463854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb779325-7f6b-4c2b-bb97-5db2d4883954-registry-certificates\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.463956 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.463905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb779325-7f6b-4c2b-bb97-5db2d4883954-trusted-ca\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.463956 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.463938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-bound-sa-token\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.464057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.463964 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-registry-tls\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.464057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.463995 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb779325-7f6b-4c2b-bb97-5db2d4883954-ca-trust-extracted\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.464057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.464018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb779325-7f6b-4c2b-bb97-5db2d4883954-installation-pull-secrets\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.464057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.464053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7wj\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-kube-api-access-wm7wj\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.464465 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.464438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cb779325-7f6b-4c2b-bb97-5db2d4883954-ca-trust-extracted\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.464837 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.464809 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cb779325-7f6b-4c2b-bb97-5db2d4883954-registry-certificates\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.464964 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.464819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb779325-7f6b-4c2b-bb97-5db2d4883954-trusted-ca\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.466399 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.466372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cb779325-7f6b-4c2b-bb97-5db2d4883954-image-registry-private-configuration\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.466586 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.466568 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 23:28:58.466645 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.466621 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cb779325-7f6b-4c2b-bb97-5db2d4883954-installation-pull-secrets\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.466770 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.466753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-registry-tls\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.472764 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.472741 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7wj\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-kube-api-access-wm7wj\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.472877 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.472866 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cb779325-7f6b-4c2b-bb97-5db2d4883954-bound-sa-token\") pod \"image-registry-547f948bc6-nmfj5\" (UID: \"cb779325-7f6b-4c2b-bb97-5db2d4883954\") " pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.476104 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.476086 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b38ad682-bd50-4725-b3c1-821e12e3e15f-original-pull-secret\") pod \"global-pull-secret-syncer-8gm96\" (UID: \"b38ad682-bd50-4725-b3c1-821e12e3e15f\") " pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:58.495274 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.495250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:58.525880 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.525845 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:28:58.564642 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.564607 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:58.566758 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.566738 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 23:28:58.578324 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.578304 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 23:28:58.589477 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.589451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t88d9\" (UniqueName: \"kubernetes.io/projected/37fa4fdd-fff0-4421-a01b-b8139858c0d4-kube-api-access-t88d9\") pod \"network-check-target-mhsqr\" (UID: \"37fa4fdd-fff0-4421-a01b-b8139858c0d4\") " pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:58.596386 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.596356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:28:58.622176 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.622147 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gjrpj\"" Jul 09 23:28:58.626252 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.626229 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-r9l9m\"" Jul 09 23:28:58.631096 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.631073 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:58.633049 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.633031 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8rpsb\"" Jul 09 23:28:58.634954 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.634935 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fz2qx" Jul 09 23:28:58.637590 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.637555 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gm96" Jul 09 23:28:58.641157 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.641129 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:28:58.648825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.648439 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95"] Jul 09 23:28:58.654879 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:28:58.654725 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3848a6a_3649_4d85_9694_c66d96c7119a.slice/crio-9a8b68d511af2a6de1b9f140b602ba36a8d81fbcdd575af5e80b6ed90475fffb WatchSource:0}: Error finding container 9a8b68d511af2a6de1b9f140b602ba36a8d81fbcdd575af5e80b6ed90475fffb: Status 404 returned error can't find the container with id 9a8b68d511af2a6de1b9f140b602ba36a8d81fbcdd575af5e80b6ed90475fffb Jul 09 23:28:58.793136 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.793082 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-547f948bc6-nmfj5"] Jul 09 23:28:58.797137 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:28:58.797108 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb779325_7f6b_4c2b_bb97_5db2d4883954.slice/crio-7ca6f62ffe3b48d6563d33a9280e9dffe00a6185590bb6927581054920fc6678 WatchSource:0}: Error finding container 7ca6f62ffe3b48d6563d33a9280e9dffe00a6185590bb6927581054920fc6678: Status 404 returned error can't find the container with id 7ca6f62ffe3b48d6563d33a9280e9dffe00a6185590bb6927581054920fc6678 Jul 09 23:28:58.807951 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:58.807721 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gm96"] Jul 09 23:28:58.811243 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:28:58.811193 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38ad682_bd50_4725_b3c1_821e12e3e15f.slice/crio-47da734f13f0a67b70d7ea2259439764992fcea02d342cd0d537d1dfdad0c904 WatchSource:0}: Error finding container 47da734f13f0a67b70d7ea2259439764992fcea02d342cd0d537d1dfdad0c904: Status 404 returned error can't find the container with id 47da734f13f0a67b70d7ea2259439764992fcea02d342cd0d537d1dfdad0c904 Jul 09 23:28:59.032372 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.032345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mhsqr"] Jul 09 23:28:59.033984 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.033675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gm96" event={"ID":"b38ad682-bd50-4725-b3c1-821e12e3e15f","Type":"ContainerStarted","Data":"47da734f13f0a67b70d7ea2259439764992fcea02d342cd0d537d1dfdad0c904"} Jul 09 23:28:59.034868 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.034843 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fz2qx"] Jul 09 23:28:59.035873 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.035795 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" event={"ID":"cb779325-7f6b-4c2b-bb97-5db2d4883954","Type":"ContainerStarted","Data":"5beaffb193ccb5158281f0b11f82f546757a0049e72661adf7b3adb34bbeb0d7"} Jul 09 23:28:59.035873 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.035842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" event={"ID":"cb779325-7f6b-4c2b-bb97-5db2d4883954","Type":"ContainerStarted","Data":"7ca6f62ffe3b48d6563d33a9280e9dffe00a6185590bb6927581054920fc6678"} Jul 09 23:28:59.036570 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.036543 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:28:59.038556 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.038531 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:59.038688 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.038580 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" event={"ID":"f3848a6a-3649-4d85-9694-c66d96c7119a","Type":"ContainerStarted","Data":"9a8b68d511af2a6de1b9f140b602ba36a8d81fbcdd575af5e80b6ed90475fffb"} Jul 09 23:28:59.040255 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:28:59.040228 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f4db34_e093_4e81_9bde_54b01feb897f.slice/crio-7fe831ef3e8e8c13cd9f531bd9d07ef4b6c1c0328f0e2c045b0f1b5e896caada WatchSource:0}: Error finding container 7fe831ef3e8e8c13cd9f531bd9d07ef4b6c1c0328f0e2c045b0f1b5e896caada: Status 404 returned error can't find the container with id 7fe831ef3e8e8c13cd9f531bd9d07ef4b6c1c0328f0e2c045b0f1b5e896caada Jul 09 23:28:59.047057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.047040 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:28:59.057423 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.057383 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" podStartSLOduration=1.0573681129999999 podStartE2EDuration="1.057368113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:28:59.056839015 +0000 UTC m=+66.937590977" watchObservedRunningTime="2026-07-09 23:28:59.057368113 +0000 UTC m=+66.938120063" Jul 09 23:28:59.169601 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.169567 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-registry-certificates\") pod \"67a109cb-dc48-4065-a414-64c89161ff14\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " Jul 09 23:28:59.169757 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.169625 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-image-registry-private-configuration\") pod \"67a109cb-dc48-4065-a414-64c89161ff14\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " Jul 09 23:28:59.169757 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.169680 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-trusted-ca\") pod \"67a109cb-dc48-4065-a414-64c89161ff14\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " Jul 09 23:28:59.169757 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.169713 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-installation-pull-secrets\") pod \"67a109cb-dc48-4065-a414-64c89161ff14\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " Jul 09 23:28:59.169910 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.169764 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-bound-sa-token\") pod \"67a109cb-dc48-4065-a414-64c89161ff14\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " Jul 09 23:28:59.169910 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.169799 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmxvz\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-kube-api-access-bmxvz\") pod \"67a109cb-dc48-4065-a414-64c89161ff14\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " Jul 09 23:28:59.169910 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.169833 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67a109cb-dc48-4065-a414-64c89161ff14-ca-trust-extracted\") pod \"67a109cb-dc48-4065-a414-64c89161ff14\" (UID: \"67a109cb-dc48-4065-a414-64c89161ff14\") " Jul 09 23:28:59.170256 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.170191 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a109cb-dc48-4065-a414-64c89161ff14-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "67a109cb-dc48-4065-a414-64c89161ff14" (UID: "67a109cb-dc48-4065-a414-64c89161ff14"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:28:59.170830 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.170796 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "67a109cb-dc48-4065-a414-64c89161ff14" (UID: "67a109cb-dc48-4065-a414-64c89161ff14"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:28:59.171135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.171111 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "67a109cb-dc48-4065-a414-64c89161ff14" (UID: "67a109cb-dc48-4065-a414-64c89161ff14"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:28:59.173027 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.172950 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "67a109cb-dc48-4065-a414-64c89161ff14" (UID: "67a109cb-dc48-4065-a414-64c89161ff14"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:28:59.173027 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.172988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "67a109cb-dc48-4065-a414-64c89161ff14" (UID: "67a109cb-dc48-4065-a414-64c89161ff14"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:28:59.175624 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.175587 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-kube-api-access-bmxvz" (OuterVolumeSpecName: "kube-api-access-bmxvz") pod "67a109cb-dc48-4065-a414-64c89161ff14" (UID: "67a109cb-dc48-4065-a414-64c89161ff14"). InnerVolumeSpecName "kube-api-access-bmxvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:28:59.176124 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.176085 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "67a109cb-dc48-4065-a414-64c89161ff14" (UID: "67a109cb-dc48-4065-a414-64c89161ff14"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:28:59.270760 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.270609 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-registry-certificates\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:28:59.270760 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.270649 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-image-registry-private-configuration\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:28:59.270760 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.270665 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/67a109cb-dc48-4065-a414-64c89161ff14-trusted-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:28:59.270760 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.270679 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/67a109cb-dc48-4065-a414-64c89161ff14-installation-pull-secrets\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:28:59.270760 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.270696 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-bound-sa-token\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:28:59.270760 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.270709 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmxvz\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-kube-api-access-bmxvz\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:28:59.270760 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:28:59.270721 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/67a109cb-dc48-4065-a414-64c89161ff14-ca-trust-extracted\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:00.043608 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.043520 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhsqr" event={"ID":"37fa4fdd-fff0-4421-a01b-b8139858c0d4","Type":"ContainerStarted","Data":"9fba946380157540402a80009fbad9cbc8039c7b08b63cc714a63af7b84a0526"} Jul 09 23:29:00.047619 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.047588 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz2qx" event={"ID":"d7f4db34-e093-4e81-9bde-54b01feb897f","Type":"ContainerStarted","Data":"7fe831ef3e8e8c13cd9f531bd9d07ef4b6c1c0328f0e2c045b0f1b5e896caada"} Jul 09 23:29:00.047734 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.047659 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59b874555-jdqsx" Jul 09 23:29:00.090832 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.090798 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59b874555-jdqsx"] Jul 09 23:29:00.096233 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.096097 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59b874555-jdqsx"] Jul 09 23:29:00.180430 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.180390 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/67a109cb-dc48-4065-a414-64c89161ff14-registry-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:00.382759 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.382672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:29:00.393954 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.393884 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c-metrics-tls\") pod \"dns-default-nsrv8\" (UID: \"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c\") " pod="openshift-dns/dns-default-nsrv8" Jul 09 23:29:00.618090 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.617888 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a109cb-dc48-4065-a414-64c89161ff14" path="/var/lib/kubelet/pods/67a109cb-dc48-4065-a414-64c89161ff14/volumes" Jul 09 23:29:00.624916 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.624717 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8548g\"" Jul 09 23:29:00.632939 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:00.632689 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nsrv8" Jul 09 23:29:03.741721 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.741687 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:03.838750 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.838725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:03.921134 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.921098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:03.923776 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.923746 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg"] Jul 09 23:29:03.927138 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.927117 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:03.929549 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.929528 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 09 23:29:03.929680 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.929587 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-qstbf\"" Jul 09 23:29:03.935375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.935309 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg"] Jul 09 23:29:03.981136 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:03.981103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:04.013317 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.013284 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45b9e056-37fd-4765-bd6d-ea3e99d04da8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f7rdg\" (UID: \"45b9e056-37fd-4765-bd6d-ea3e99d04da8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:04.114787 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.114749 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45b9e056-37fd-4765-bd6d-ea3e99d04da8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f7rdg\" (UID: \"45b9e056-37fd-4765-bd6d-ea3e99d04da8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:04.114972 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:04.114921 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 09 23:29:04.115040 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:04.115013 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45b9e056-37fd-4765-bd6d-ea3e99d04da8-tls-certificates podName:45b9e056-37fd-4765-bd6d-ea3e99d04da8 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:04.614991085 +0000 UTC m=+72.495743043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/45b9e056-37fd-4765-bd6d-ea3e99d04da8-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-f7rdg" (UID: "45b9e056-37fd-4765-bd6d-ea3e99d04da8") : secret "prometheus-operator-admission-webhook-tls" not found Jul 09 23:29:04.175034 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.175009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:04.272810 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.272713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:04.372820 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.372784 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:04.620225 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.620127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45b9e056-37fd-4765-bd6d-ea3e99d04da8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f7rdg\" (UID: \"45b9e056-37fd-4765-bd6d-ea3e99d04da8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:04.623101 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.623076 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/45b9e056-37fd-4765-bd6d-ea3e99d04da8-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-f7rdg\" (UID: \"45b9e056-37fd-4765-bd6d-ea3e99d04da8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:04.654310 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.654256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:04.838025 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:04.837981 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:05.457678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:05.457653 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nsrv8"] Jul 09 23:29:05.466507 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:05.462863 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d4fc34_87c4_479c_9b9e_dc7e6d4b4b1c.slice/crio-2a4f1e7ed3986fe664185276963c2965f9479923f864df51051168fa04fd7fbb WatchSource:0}: Error finding container 2a4f1e7ed3986fe664185276963c2965f9479923f864df51051168fa04fd7fbb: Status 404 returned error can't find the container with id 2a4f1e7ed3986fe664185276963c2965f9479923f864df51051168fa04fd7fbb Jul 09 23:29:05.478973 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:05.478847 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg"] Jul 09 23:29:06.068882 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.068842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mhsqr" event={"ID":"37fa4fdd-fff0-4421-a01b-b8139858c0d4","Type":"ContainerStarted","Data":"1069f1a6aa055323f228eb95919949d5679e5edc884c6ee844d1cb51dfcbffb4"} Jul 09 23:29:06.069358 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.068956 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:29:06.071130 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.071100 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz2qx" event={"ID":"d7f4db34-e093-4e81-9bde-54b01feb897f","Type":"ContainerStarted","Data":"d470c2aeeeecd9381c81dea0ab7d2064aa383dfb9741c13ae473b32924546496"} Jul 09 23:29:06.071284 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.071157 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fz2qx" event={"ID":"d7f4db34-e093-4e81-9bde-54b01feb897f","Type":"ContainerStarted","Data":"e3f3258160b80e5182622917052fa528089f75d38d42ad3a21ba4766636352df"} Jul 09 23:29:06.072337 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.072312 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsrv8" event={"ID":"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c","Type":"ContainerStarted","Data":"2a4f1e7ed3986fe664185276963c2965f9479923f864df51051168fa04fd7fbb"} Jul 09 23:29:06.073974 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.073946 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gm96" event={"ID":"b38ad682-bd50-4725-b3c1-821e12e3e15f","Type":"ContainerStarted","Data":"dc25fa8a6d7b81ff85e0ec4bb4d5c737b4f70bc6b289999cbf6b958f10d71660"} Jul 09 23:29:06.075576 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.075547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" event={"ID":"f3848a6a-3649-4d85-9694-c66d96c7119a","Type":"ContainerStarted","Data":"23134f6d63968534d6890ff33aa7ac1dc2f5320996eb7586435d8aa3d003b625"} Jul 09 23:29:06.075692 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.075655 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:29:06.076710 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.076688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" event={"ID":"45b9e056-37fd-4765-bd6d-ea3e99d04da8","Type":"ContainerStarted","Data":"11b0bacd270de5016f987c4e2e10abf83fc1d3314758510ed3827455408d6fff"} Jul 09 23:29:06.077457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.077433 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" Jul 09 23:29:06.083025 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.082980 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mhsqr" podStartSLOduration=66.819917181 podStartE2EDuration="1m13.082955034s" totalImagesPullingTime="6.263037853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:06.082782517 +0000 UTC m=+73.963534480" watchObservedRunningTime="2026-07-09 23:29:06.082955034 +0000 UTC m=+73.963706989" Jul 09 23:29:06.096879 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.096825 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58bfc74fd-4gr95" podStartSLOduration=1.460333093 podStartE2EDuration="8.0968053s" totalImagesPullingTime="6.636472207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:06.096802676 +0000 UTC m=+73.977554635" watchObservedRunningTime="2026-07-09 23:29:06.0968053 +0000 UTC m=+73.977557242" Jul 09 23:29:06.114671 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.114620 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fz2qx" podStartSLOduration=66.86679249 podStartE2EDuration="1m13.114583621s" totalImagesPullingTime="6.247791131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:06.113399469 +0000 UTC m=+73.994151429" watchObservedRunningTime="2026-07-09 23:29:06.114583621 +0000 UTC m=+73.995335584" Jul 09 23:29:06.127933 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.127886 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8gm96" podStartSLOduration=66.632986826 podStartE2EDuration="1m13.12787108s" totalImagesPullingTime="6.494884254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:06.126704741 +0000 UTC m=+74.007456702" watchObservedRunningTime="2026-07-09 23:29:06.12787108 +0000 UTC m=+74.008623040" Jul 09 23:29:06.194445 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.194359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:06.243797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:06.243763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:07.865784 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:07.865751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:07.965801 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:07.965771 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:08.047439 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.047405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:29:08.049828 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.049799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a383ff5-7316-487d-80c8-93f4ca9c6fde-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwjxn\" (UID: \"0a383ff5-7316-487d-80c8-93f4ca9c6fde\") " pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:29:08.086613 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.086580 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsrv8" event={"ID":"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c","Type":"ContainerStarted","Data":"310f5da5117cd8a7bede2732691e470067210cff0d413bd9ff35427fe18fbd9b"} Jul 09 23:29:08.086769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.086669 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsrv8" event={"ID":"a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c","Type":"ContainerStarted","Data":"48a1d879f977722c77f04f17a1ed05d515afd476196dbcf0f026d8f24b36aa7f"} Jul 09 23:29:08.086769 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.086703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nsrv8" Jul 09 23:29:08.088029 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.087996 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" event={"ID":"45b9e056-37fd-4765-bd6d-ea3e99d04da8","Type":"ContainerStarted","Data":"9c5773a263c014669d359abb793dea2c79f0e4a6c5e4d367d3c16ac42a7216e7"} Jul 09 23:29:08.101825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.101781 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nsrv8" podStartSLOduration=38.341063892 podStartE2EDuration="40.101769491s" totalImagesPullingTime="1.760705599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:08.101404878 +0000 UTC m=+75.982156837" watchObservedRunningTime="2026-07-09 23:29:08.101769491 +0000 UTC m=+75.982521452" Jul 09 23:29:08.116004 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.115924 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" podStartSLOduration=3.382181846 podStartE2EDuration="5.115908707s" totalImagesPullingTime="1.733726861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:08.114661023 +0000 UTC m=+75.995412984" watchObservedRunningTime="2026-07-09 23:29:08.115908707 +0000 UTC m=+75.996660666" Jul 09 23:29:08.218599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.218567 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qktl8\"" Jul 09 23:29:08.227434 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.227408 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwjxn" Jul 09 23:29:08.350220 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:08.350161 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwjxn"] Jul 09 23:29:08.354078 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:08.354052 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a383ff5_7316_487d_80c8_93f4ca9c6fde.slice/crio-cd1c8fa98241ab0eef1672bab3e04732de43c589cdf0b0512e54710dfe3494d6 WatchSource:0}: Error finding container cd1c8fa98241ab0eef1672bab3e04732de43c589cdf0b0512e54710dfe3494d6: Status 404 returned error can't find the container with id cd1c8fa98241ab0eef1672bab3e04732de43c589cdf0b0512e54710dfe3494d6 Jul 09 23:29:09.094886 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:09.094783 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwjxn" event={"ID":"0a383ff5-7316-487d-80c8-93f4ca9c6fde","Type":"ContainerStarted","Data":"cd1c8fa98241ab0eef1672bab3e04732de43c589cdf0b0512e54710dfe3494d6"} Jul 09 23:29:09.095968 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:09.095668 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:09.101998 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:09.101798 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-f7rdg" Jul 09 23:29:09.868446 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:09.868412 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:09.971350 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:09.971320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:10.003039 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.003000 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x6w6j"] Jul 09 23:29:10.010497 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.010475 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.014985 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.014963 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 23:29:10.015077 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.014963 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 23:29:10.016639 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.016614 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 23:29:10.016742 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.016675 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 23:29:10.016742 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.016681 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 23:29:10.016841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.016674 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kpjr2\"" Jul 09 23:29:10.026818 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.026794 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x6w6j"] Jul 09 23:29:10.047155 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.047117 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:10.063246 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.063213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.063401 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.063259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.063401 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.063302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxtd2\" (UniqueName: \"kubernetes.io/projected/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-kube-api-access-vxtd2\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.063501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.063411 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.164087 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.164005 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.164087 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.164055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.164087 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.164093 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxtd2\" (UniqueName: \"kubernetes.io/projected/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-kube-api-access-vxtd2\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.164688 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.164133 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.164688 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:10.164263 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 09 23:29:10.164688 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:10.164329 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls podName:2b4c2f26-5ff9-49b3-91fa-729707e7a2eb nodeName:}" failed. No retries permitted until 2026-07-09 23:29:10.664312444 +0000 UTC m=+78.545064382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls") pod "prometheus-operator-64785b6675-x6w6j" (UID: "2b4c2f26-5ff9-49b3-91fa-729707e7a2eb") : secret "prometheus-operator-tls" not found Jul 09 23:29:10.164923 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.164899 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.167075 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.167052 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.172876 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.172846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxtd2\" (UniqueName: \"kubernetes.io/projected/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-kube-api-access-vxtd2\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.665445 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.665410 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-848bc7b456-hwrpb"] Jul 09 23:29:10.668357 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.668322 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:10.668547 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:10.668529 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 09 23:29:10.668645 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:10.668590 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls podName:2b4c2f26-5ff9-49b3-91fa-729707e7a2eb nodeName:}" failed. No retries permitted until 2026-07-09 23:29:11.668573047 +0000 UTC m=+79.549324990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls") pod "prometheus-operator-64785b6675-x6w6j" (UID: "2b4c2f26-5ff9-49b3-91fa-729707e7a2eb") : secret "prometheus-operator-tls" not found Jul 09 23:29:10.680021 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.679995 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.683386 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.683305 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 23:29:10.683386 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.683373 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 23:29:10.683867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.683518 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 23:29:10.683867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.683551 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 23:29:10.683867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.683611 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848bc7b456-hwrpb"] Jul 09 23:29:10.684024 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.683994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:10.684272 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.684139 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 23:29:10.684477 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.684460 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m97q4\"" Jul 09 23:29:10.684612 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.684527 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 23:29:10.684743 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.684726 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 23:29:10.689242 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.688752 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 23:29:10.735977 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.735941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:10.769086 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.769050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-serving-cert\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.769293 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.769109 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-oauth-config\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.769293 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.769173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-console-config\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.769293 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.769223 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-service-ca\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.769293 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.769242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwmd7\" (UniqueName: \"kubernetes.io/projected/20c269e9-7523-475b-ac55-bfd29c5d9944-kube-api-access-jwmd7\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.769293 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.769274 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-oauth-serving-cert\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.769508 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.769376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-trusted-ca-bundle\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870024 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.869979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-trusted-ca-bundle\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.870052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-serving-cert\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.870081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-oauth-config\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.870097 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-console-config\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.870119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-service-ca\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.870143 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwmd7\" (UniqueName: \"kubernetes.io/projected/20c269e9-7523-475b-ac55-bfd29c5d9944-kube-api-access-jwmd7\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.870177 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-oauth-serving-cert\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.870966 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.870936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-trusted-ca-bundle\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.871153 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.871135 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-console-config\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.871292 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.871271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-oauth-serving-cert\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.871764 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.871744 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-service-ca\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.872848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.872827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-serving-cert\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.873157 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.873139 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-oauth-config\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.877979 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.877956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwmd7\" (UniqueName: \"kubernetes.io/projected/20c269e9-7523-475b-ac55-bfd29c5d9944-kube-api-access-jwmd7\") pod \"console-848bc7b456-hwrpb\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:10.992940 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:10.992859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:11.024038 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:11.024009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:11.677720 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:11.677685 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:11.680494 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:11.680464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/2b4c2f26-5ff9-49b3-91fa-729707e7a2eb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x6w6j\" (UID: \"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:11.822559 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:11.822522 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" Jul 09 23:29:12.188413 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:12.188388 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x6w6j"] Jul 09 23:29:12.191810 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:12.191740 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4c2f26_5ff9_49b3_91fa_729707e7a2eb.slice/crio-08738dc579cd54d2a72d1e4bf016ee1e90e3d494a1b80facb90e5132cd9e19ce WatchSource:0}: Error finding container 08738dc579cd54d2a72d1e4bf016ee1e90e3d494a1b80facb90e5132cd9e19ce: Status 404 returned error can't find the container with id 08738dc579cd54d2a72d1e4bf016ee1e90e3d494a1b80facb90e5132cd9e19ce Jul 09 23:29:12.197712 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:12.197687 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848bc7b456-hwrpb"] Jul 09 23:29:12.200839 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:12.200804 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c269e9_7523_475b_ac55_bfd29c5d9944.slice/crio-4ad0cc730a268024eab8c31b9b91cd13bdc762013f438bc8071c91a5327635f1 WatchSource:0}: Error finding container 4ad0cc730a268024eab8c31b9b91cd13bdc762013f438bc8071c91a5327635f1: Status 404 returned error can't find the container with id 4ad0cc730a268024eab8c31b9b91cd13bdc762013f438bc8071c91a5327635f1 Jul 09 23:29:13.114183 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:13.114138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwjxn" event={"ID":"0a383ff5-7316-487d-80c8-93f4ca9c6fde","Type":"ContainerStarted","Data":"ddc38ae4a615a14c3bd1c07847fba90e8a12eb9cd8dede3557c6a09856dd397c"} Jul 09 23:29:13.114632 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:13.114253 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwjxn" event={"ID":"0a383ff5-7316-487d-80c8-93f4ca9c6fde","Type":"ContainerStarted","Data":"699a9b464bcd6f1c626e850f6f78dafde43f8aa0ef946b7567cead2e6f5102b0"} Jul 09 23:29:13.115701 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:13.115674 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc7b456-hwrpb" event={"ID":"20c269e9-7523-475b-ac55-bfd29c5d9944","Type":"ContainerStarted","Data":"4ad0cc730a268024eab8c31b9b91cd13bdc762013f438bc8071c91a5327635f1"} Jul 09 23:29:13.117144 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:13.117114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" event={"ID":"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb","Type":"ContainerStarted","Data":"08738dc579cd54d2a72d1e4bf016ee1e90e3d494a1b80facb90e5132cd9e19ce"} Jul 09 23:29:13.914399 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:13.914314 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:14.123596 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:14.123556 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" event={"ID":"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb","Type":"ContainerStarted","Data":"a90777e086cf4c5f5bddd45e9e2151705968a44a0d04bc7e38919bdb7b5d2e8c"} Jul 09 23:29:14.124046 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:14.123614 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" event={"ID":"2b4c2f26-5ff9-49b3-91fa-729707e7a2eb","Type":"ContainerStarted","Data":"4b6735120fa6d1898668a9619dcde1f37595953b6cdac2e4a2b78a0d915882d1"} Jul 09 23:29:14.139113 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:14.139054 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-x6w6j" podStartSLOduration=3.725727264 podStartE2EDuration="5.139033491s" totalImagesPullingTime="1.413306227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:14.138729817 +0000 UTC m=+82.019481778" watchObservedRunningTime="2026-07-09 23:29:14.139033491 +0000 UTC m=+82.019785431" Jul 09 23:29:15.040717 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:15.040686 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:15.092481 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:15.092441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:16.094335 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.094299 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:16.306561 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.306527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:16.349832 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.349755 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q4mks"] Jul 09 23:29:16.353033 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.353013 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.355617 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.355594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 23:29:16.355844 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.355829 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 23:29:16.356310 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.356297 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mhc8b\"" Jul 09 23:29:16.356522 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.356510 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 23:29:16.398512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.398461 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:16.419570 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-wtmp\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419710 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrlpd\" (UniqueName: \"kubernetes.io/projected/1d7f8352-3225-4ae2-9596-edfcee60dda0-kube-api-access-wrlpd\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419710 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419710 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419710 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419653 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-textfile\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419710 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f8352-3225-4ae2-9596-edfcee60dda0-metrics-client-ca\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419865 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419713 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-sys\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419865 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419734 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-root\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.419865 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.419751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.476936 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.476898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:16.520310 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520267 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f8352-3225-4ae2-9596-edfcee60dda0-metrics-client-ca\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520322 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-sys\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520387 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-sys\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520416 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-root\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520490 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-root\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-wtmp\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlpd\" (UniqueName: \"kubernetes.io/projected/1d7f8352-3225-4ae2-9596-edfcee60dda0-kube-api-access-wrlpd\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520797 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:16.520619 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 23:29:16.520797 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:16.520680 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls podName:1d7f8352-3225-4ae2-9596-edfcee60dda0 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:17.020659618 +0000 UTC m=+84.901411571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls") pod "node-exporter-q4mks" (UID: "1d7f8352-3225-4ae2-9596-edfcee60dda0") : secret "node-exporter-tls" not found Jul 09 23:29:16.520797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520622 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-wtmp\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.520797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.521143 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-textfile\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.521143 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.520895 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d7f8352-3225-4ae2-9596-edfcee60dda0-metrics-client-ca\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.521143 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.521115 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-textfile\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.521509 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.521491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-accelerators-collector-config\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.523126 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.523109 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.530481 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.530460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlpd\" (UniqueName: \"kubernetes.io/projected/1d7f8352-3225-4ae2-9596-edfcee60dda0-kube-api-access-wrlpd\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:16.540566 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.540544 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:16.706566 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.706488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:16.806682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:16.806652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.025914 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.025871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:17.026051 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:17.026012 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 09 23:29:17.026099 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:17.026072 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls podName:1d7f8352-3225-4ae2-9596-edfcee60dda0 nodeName:}" failed. No retries permitted until 2026-07-09 23:29:18.026055978 +0000 UTC m=+85.906807919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls") pod "node-exporter-q4mks" (UID: "1d7f8352-3225-4ae2-9596-edfcee60dda0") : secret "node-exporter-tls" not found Jul 09 23:29:17.098581 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.098545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.136481 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.136440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwjxn" event={"ID":"0a383ff5-7316-487d-80c8-93f4ca9c6fde","Type":"ContainerStarted","Data":"7a9abe9310741e52c15433d604f2d0e07dce2d6ea7de700cb2103c67ca855a58"} Jul 09 23:29:17.137969 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.137950 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc7b456-hwrpb" event={"ID":"20c269e9-7523-475b-ac55-bfd29c5d9944","Type":"ContainerStarted","Data":"06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad"} Jul 09 23:29:17.153988 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.153764 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gwjxn" podStartSLOduration=33.458425496 podStartE2EDuration="41.153745802s" totalImagesPullingTime="7.695320306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:28:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:17.153122381 +0000 UTC m=+85.033874342" watchObservedRunningTime="2026-07-09 23:29:17.153745802 +0000 UTC m=+85.034497766" Jul 09 23:29:17.162015 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.161992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.169765 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.169727 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-848bc7b456-hwrpb" podStartSLOduration=3.167198168 podStartE2EDuration="7.169712958s" totalImagesPullingTime="4.00251479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:17.168664915 +0000 UTC m=+85.049416877" watchObservedRunningTime="2026-07-09 23:29:17.169712958 +0000 UTC m=+85.050464917" Jul 09 23:29:17.213056 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.213034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.313516 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.313441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.394389 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.394351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.401324 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.401294 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:29:17.406189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.406168 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.408659 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.408634 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 23:29:17.408795 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.408641 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.408855 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.409132 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.409354 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.409702 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.409897 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.410090 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqnr6\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.410294 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 23:29:17.410841 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.410450 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 23:29:17.417512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.417493 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:29:17.450561 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.450529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.530128 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530128 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530130 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530342 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530342 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530260 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-web-config\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530342 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530300 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530342 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530469 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-volume\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530469 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530393 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530469 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530469 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530590 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gc5\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-kube-api-access-59gc5\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530590 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.530590 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.530547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-out\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.614803 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.614723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:17.631073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-web-config\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631158 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-volume\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631321 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gc5\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-kube-api-access-59gc5\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-out\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:17.631519 2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:17.631588 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls podName:6707c747-a7e4-428b-8ba6-c8b91fe7e94d nodeName:}" failed. No retries permitted until 2026-07-09 23:29:18.131567545 +0000 UTC m=+86.012319486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d") : secret "alertmanager-main-tls" not found Jul 09 23:29:17.631648 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.631587 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.632954 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.632848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.633756 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:17.633155 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle podName:6707c747-a7e4-428b-8ba6-c8b91fe7e94d nodeName:}" failed. No retries permitted until 2026-07-09 23:29:18.133133634 +0000 UTC m=+86.013885596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d") : configmap references non-existent config key: ca-bundle.crt Jul 09 23:29:17.634475 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.634430 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-out\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.634577 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.634536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.634710 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.634675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-web-config\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.635103 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.635065 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.635307 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.635287 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.635427 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.635353 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-volume\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.635669 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.635652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.635852 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.635833 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.641030 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.641010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gc5\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-kube-api-access-59gc5\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:17.714631 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:17.714584 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:18.034413 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.034384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:18.036804 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.036774 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1d7f8352-3225-4ae2-9596-edfcee60dda0-node-exporter-tls\") pod \"node-exporter-q4mks\" (UID: \"1d7f8352-3225-4ae2-9596-edfcee60dda0\") " pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:18.097610 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.097579 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nsrv8" Jul 09 23:29:18.135796 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.135750 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:18.136246 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.135834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:18.136991 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.136967 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:18.138532 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.138511 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:18.157448 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.157415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:18.162417 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.162396 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q4mks" Jul 09 23:29:18.174075 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:18.174050 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7f8352_3225_4ae2_9596_edfcee60dda0.slice/crio-ca93edafacaf63dc4ab0a7672f3ce69f49fb4699ae79bb28043a1e65b33d5ce9 WatchSource:0}: Error finding container ca93edafacaf63dc4ab0a7672f3ce69f49fb4699ae79bb28043a1e65b33d5ce9: Status 404 returned error can't find the container with id ca93edafacaf63dc4ab0a7672f3ce69f49fb4699ae79bb28043a1e65b33d5ce9 Jul 09 23:29:18.256814 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.256779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:18.316849 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.316765 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:29:18.380451 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.380398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:18.453832 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.453808 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:29:18.454790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:18.454762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:18.456295 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:18.456271 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6707c747_a7e4_428b_8ba6_c8b91fe7e94d.slice/crio-6b4c810dfd6b7502e8ef570a7cb1fe2c26d7589780730d837b59030b71672964 WatchSource:0}: Error finding container 6b4c810dfd6b7502e8ef570a7cb1fe2c26d7589780730d837b59030b71672964: Status 404 returned error can't find the container with id 6b4c810dfd6b7502e8ef570a7cb1fe2c26d7589780730d837b59030b71672964 Jul 09 23:29:19.148022 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:19.147981 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4mks" event={"ID":"1d7f8352-3225-4ae2-9596-edfcee60dda0","Type":"ContainerStarted","Data":"ca93edafacaf63dc4ab0a7672f3ce69f49fb4699ae79bb28043a1e65b33d5ce9"} Jul 09 23:29:19.149564 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:19.149532 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerStarted","Data":"6b4c810dfd6b7502e8ef570a7cb1fe2c26d7589780730d837b59030b71672964"} Jul 09 23:29:19.257302 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:19.257275 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:19.359396 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:19.359335 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:19.463926 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:19.463876 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:19.610456 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:19.610415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:20.155567 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:20.155532 2562 generic.go:356] "Generic (PLEG): container finished" podID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerID="d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808" exitCode=0 Jul 09 23:29:20.155980 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:20.155620 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808"} Jul 09 23:29:20.157318 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:20.157255 2562 generic.go:356] "Generic (PLEG): container finished" podID="1d7f8352-3225-4ae2-9596-edfcee60dda0" containerID="80ba33c79e3221de8bd36ca86641f23983fd58f58d43756b40e938407d0aa47e" exitCode=0 Jul 09 23:29:20.157384 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:20.157318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4mks" event={"ID":"1d7f8352-3225-4ae2-9596-edfcee60dda0","Type":"ContainerDied","Data":"80ba33c79e3221de8bd36ca86641f23983fd58f58d43756b40e938407d0aa47e"} Jul 09 23:29:20.993574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:20.993536 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:20.993765 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:20.993612 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:20.999812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:20.999794 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:21.055825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.055796 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-547f948bc6-nmfj5" Jul 09 23:29:21.130514 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.130476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:21.145921 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.145870 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848bc7b456-hwrpb"] Jul 09 23:29:21.164357 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.164315 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4mks" event={"ID":"1d7f8352-3225-4ae2-9596-edfcee60dda0","Type":"ContainerStarted","Data":"477eec63f7244a1a385a85596dd7062d2bcc2849463aa8b1fed8de11403297e4"} Jul 09 23:29:21.164773 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.164386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q4mks" event={"ID":"1d7f8352-3225-4ae2-9596-edfcee60dda0","Type":"ContainerStarted","Data":"18c66278baa0da854c7d57b7d402e5333723ef23c567c13042368b4e918b6100"} Jul 09 23:29:21.169509 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.169483 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:21.176107 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.176084 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-567bfcdcdb-mpxxn"] Jul 09 23:29:21.179320 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.179302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.183611 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.183574 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q4mks" podStartSLOduration=3.997237654 podStartE2EDuration="5.183561551s" totalImagesPullingTime="1.107794481s" totalInitContainerRuntime="78.529416ms" isStatefulPod=true podCreationTimestamp="2026-07-09 23:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:21.182837662 +0000 UTC m=+89.063589621" watchObservedRunningTime="2026-07-09 23:29:21.183561551 +0000 UTC m=+89.064313511" Jul 09 23:29:21.187800 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.187780 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567bfcdcdb-mpxxn"] Jul 09 23:29:21.211547 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.211512 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:21.261592 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.261566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:21.262737 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.262706 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-trusted-ca-bundle\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.263093 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.263073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-service-ca\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.263173 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.263142 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-oauth-serving-cert\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.263260 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.263243 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-serving-cert\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.263325 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.263277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9wvv\" (UniqueName: \"kubernetes.io/projected/a2c63b46-347f-474f-a071-9c731682a89c-kube-api-access-g9wvv\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.263427 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.263403 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-oauth-config\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.263480 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.263445 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-console-config\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364147 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-trusted-ca-bundle\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-service-ca\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-oauth-serving-cert\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-serving-cert\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9wvv\" (UniqueName: \"kubernetes.io/projected/a2c63b46-347f-474f-a071-9c731682a89c-kube-api-access-g9wvv\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-oauth-config\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-console-config\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.364998 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.364960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-service-ca\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.365135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.365069 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-console-config\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.365237 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.365173 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-trusted-ca-bundle\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.365559 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.365537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-oauth-serving-cert\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.367586 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.367565 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-serving-cert\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.368409 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.368385 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-oauth-config\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.372797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.372767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9wvv\" (UniqueName: \"kubernetes.io/projected/a2c63b46-347f-474f-a071-9c731682a89c-kube-api-access-g9wvv\") pod \"console-567bfcdcdb-mpxxn\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.489744 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.489697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:21.575414 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.575350 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dc7c784b8-9slq8"] Jul 09 23:29:21.578597 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.578580 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.581402 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.581376 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 09 23:29:21.581527 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.581432 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 09 23:29:21.581623 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.581605 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 09 23:29:21.581790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.581765 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s6f28\"" Jul 09 23:29:21.581917 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.581889 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 09 23:29:21.582002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.581922 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 09 23:29:21.586586 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.586566 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 09 23:29:21.592169 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.592141 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc7c784b8-9slq8"] Jul 09 23:29:21.614389 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.614361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:21.666919 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.666820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-serving-certs-ca-bundle\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.666919 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.666861 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.666919 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.666894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-metrics-client-ca\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.667140 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.666979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-secret-telemeter-client\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.667140 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.667011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-telemeter-client-tls\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.667140 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.667035 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8wl\" (UniqueName: \"kubernetes.io/projected/88e3646d-e876-4e03-94a8-2917f3fbeec8-kube-api-access-vp8wl\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.667140 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.667060 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-federate-client-tls\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.667345 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.667141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.674254 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.674223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:21.717424 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.717402 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-567bfcdcdb-mpxxn"] Jul 09 23:29:21.719694 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:21.719669 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c63b46_347f_474f_a071_9c731682a89c.slice/crio-48d2522fc6648e16e2ade1079a97cc25864d089161bf8a2f9374dab530dee70d WatchSource:0}: Error finding container 48d2522fc6648e16e2ade1079a97cc25864d089161bf8a2f9374dab530dee70d: Status 404 returned error can't find the container with id 48d2522fc6648e16e2ade1079a97cc25864d089161bf8a2f9374dab530dee70d Jul 09 23:29:21.767676 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.767652 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-serving-certs-ca-bundle\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.768416 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.767698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.768538 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.768465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-metrics-client-ca\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.768538 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.768505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-secret-telemeter-client\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.768652 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.768537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-telemeter-client-tls\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.768652 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.768563 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8wl\" (UniqueName: \"kubernetes.io/projected/88e3646d-e876-4e03-94a8-2917f3fbeec8-kube-api-access-vp8wl\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.768652 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.768590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-federate-client-tls\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.768652 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.768623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.770005 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.768921 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-serving-certs-ca-bundle\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.770005 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.769503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.770005 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.769952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88e3646d-e876-4e03-94a8-2917f3fbeec8-metrics-client-ca\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.771084 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.771040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.772450 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.772377 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-secret-telemeter-client\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.772450 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.772386 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-federate-client-tls\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.772845 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.772821 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/88e3646d-e876-4e03-94a8-2917f3fbeec8-telemeter-client-tls\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.777230 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.777191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8wl\" (UniqueName: \"kubernetes.io/projected/88e3646d-e876-4e03-94a8-2917f3fbeec8-kube-api-access-vp8wl\") pod \"telemeter-client-6dc7c784b8-9slq8\" (UID: \"88e3646d-e876-4e03-94a8-2917f3fbeec8\") " pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.901683 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.901660 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" Jul 09 23:29:21.913260 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:21.913171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:22.013481 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.013448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:22.044084 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.044016 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc7c784b8-9slq8"] Jul 09 23:29:22.048164 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:22.048138 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e3646d_e876_4e03_94a8_2917f3fbeec8.slice/crio-091ad4023a21e23b291f614f9a9da7c21af0acf8ce91dd56dea4eb72608c1083 WatchSource:0}: Error finding container 091ad4023a21e23b291f614f9a9da7c21af0acf8ce91dd56dea4eb72608c1083: Status 404 returned error can't find the container with id 091ad4023a21e23b291f614f9a9da7c21af0acf8ce91dd56dea4eb72608c1083 Jul 09 23:29:22.113613 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.113522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:22.169701 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.169667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:22.172809 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.172787 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerStarted","Data":"b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c"} Jul 09 23:29:22.172925 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.172884 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerStarted","Data":"25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8"} Jul 09 23:29:22.172925 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.172919 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerStarted","Data":"02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4"} Jul 09 23:29:22.173011 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.172947 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerStarted","Data":"47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3"} Jul 09 23:29:22.173011 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.172974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerStarted","Data":"6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31"} Jul 09 23:29:22.174158 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.174129 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bfcdcdb-mpxxn" event={"ID":"a2c63b46-347f-474f-a071-9c731682a89c","Type":"ContainerStarted","Data":"e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0"} Jul 09 23:29:22.174259 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.174164 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bfcdcdb-mpxxn" event={"ID":"a2c63b46-347f-474f-a071-9c731682a89c","Type":"ContainerStarted","Data":"48d2522fc6648e16e2ade1079a97cc25864d089161bf8a2f9374dab530dee70d"} Jul 09 23:29:22.175246 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.175226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" event={"ID":"88e3646d-e876-4e03-94a8-2917f3fbeec8","Type":"ContainerStarted","Data":"091ad4023a21e23b291f614f9a9da7c21af0acf8ce91dd56dea4eb72608c1083"} Jul 09 23:29:22.192565 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.192527 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-567bfcdcdb-mpxxn" podStartSLOduration=1.192514188 podStartE2EDuration="1.192514188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:22.191898277 +0000 UTC m=+90.072650236" watchObservedRunningTime="2026-07-09 23:29:22.192514188 +0000 UTC m=+90.073266148" Jul 09 23:29:22.512116 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.512080 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:22.521894 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.521854 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:29:22.527687 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.526613 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.530423 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530387 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 23:29:22.530561 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530438 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 23:29:22.530561 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530457 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 23:29:22.530561 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530387 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 23:29:22.530753 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530736 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 23:29:22.530888 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530873 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 23:29:22.530950 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530908 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 23:29:22.530950 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530738 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 23:29:22.531031 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.530877 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 23:29:22.531134 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.531120 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9o6nu3iaunbns\"" Jul 09 23:29:22.531329 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.531314 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 23:29:22.531466 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.531452 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 23:29:22.531549 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.531498 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 23:29:22.531775 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.531610 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pt44r\"" Jul 09 23:29:22.531775 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.531709 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 23:29:22.535844 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.535667 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 23:29:22.536588 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.536409 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 23:29:22.542692 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.542473 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:29:22.677139 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677099 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677158 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-config\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677410 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677436 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677487 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cflp6\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-kube-api-access-cflp6\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677568 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.677817 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.677785 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778273 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778273 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778238 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-config\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778329 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778353 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778440 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cflp6\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-kube-api-access-cflp6\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778894 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778894 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778594 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778894 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.778894 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.778646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.779083 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.779246 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.780066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779351 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.780066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.780066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.780066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779462 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.780066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779487 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.780066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.779528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.781962 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.781936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.783858 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.782639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.783858 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.782794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.784709 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.784677 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.785659 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.785228 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.785824 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.785798 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.785966 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.785934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.789062 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.788407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.789062 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.788589 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.789062 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.788906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.789308 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.789285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.789424 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.789401 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-config\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.789805 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.789785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cflp6\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-kube-api-access-cflp6\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.789805 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.789798 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.790192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.790174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.791707 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.791069 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.800315 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.800280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.802472 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.802448 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:22.840500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:22.840471 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:23.017850 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:23.017817 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:29:23.025273 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:29:23.025241 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0016e619_ebdd_46de_92e6_281d4a75071c.slice/crio-daaabc41d0ef79274b72e5b8f05fa9726b10ee5e317267005cfd05f1b5d75413 WatchSource:0}: Error finding container daaabc41d0ef79274b72e5b8f05fa9726b10ee5e317267005cfd05f1b5d75413: Status 404 returned error can't find the container with id daaabc41d0ef79274b72e5b8f05fa9726b10ee5e317267005cfd05f1b5d75413 Jul 09 23:29:23.184528 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:23.184491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerStarted","Data":"86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67"} Jul 09 23:29:23.186531 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:23.186500 2562 generic.go:356] "Generic (PLEG): container finished" podID="0016e619-ebdd-46de-92e6-281d4a75071c" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" exitCode=0 Jul 09 23:29:23.186673 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:23.186569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92"} Jul 09 23:29:23.186673 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:23.186643 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerStarted","Data":"daaabc41d0ef79274b72e5b8f05fa9726b10ee5e317267005cfd05f1b5d75413"} Jul 09 23:29:23.211998 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:23.211944 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.649813992 podStartE2EDuration="6.211929602s" totalImagesPullingTime="3.546808986s" totalInitContainerRuntime="15.306624ms" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:23.210218293 +0000 UTC m=+91.090970253" watchObservedRunningTime="2026-07-09 23:29:23.211929602 +0000 UTC m=+91.092681597" Jul 09 23:29:25.199167 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:25.199128 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" event={"ID":"88e3646d-e876-4e03-94a8-2917f3fbeec8","Type":"ContainerStarted","Data":"257c75d466659c4bc7eee98c8cf5e0c43f44f418425214bdfe63ad8f3818b92f"} Jul 09 23:29:25.199599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:25.199225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" event={"ID":"88e3646d-e876-4e03-94a8-2917f3fbeec8","Type":"ContainerStarted","Data":"2efdd9f640d5117f87c6e1aca090af1b2b5ec138f6ee91819a384ea3c18e633e"} Jul 09 23:29:25.199599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:25.199259 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" event={"ID":"88e3646d-e876-4e03-94a8-2917f3fbeec8","Type":"ContainerStarted","Data":"ab7047797e1b804fb40c4cd3458ebf4fe314a407be101093b938b1a41b864988"} Jul 09 23:29:25.220099 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:25.220049 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dc7c784b8-9slq8" podStartSLOduration=2.016573545 podStartE2EDuration="4.220028729s" totalImagesPullingTime="2.203455184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:25.217868199 +0000 UTC m=+93.098620161" watchObservedRunningTime="2026-07-09 23:29:25.220028729 +0000 UTC m=+93.100780691" Jul 09 23:29:25.785064 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:25.784335 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567bfcdcdb-mpxxn"] Jul 09 23:29:25.979250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:25.979180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:26.028518 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:26.028468 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:26.575440 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:26.575399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:26.620945 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:26.620911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:26.837066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:26.836995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:26.941279 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:26.941237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:27.199153 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:27.199054 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:27.210253 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:27.210225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerStarted","Data":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} Jul 09 23:29:27.210368 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:27.210290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerStarted","Data":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} Jul 09 23:29:27.299832 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:27.299800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:29.222083 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:29.222040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerStarted","Data":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} Jul 09 23:29:29.222509 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:29.222130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerStarted","Data":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} Jul 09 23:29:29.222509 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:29.222172 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerStarted","Data":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} Jul 09 23:29:29.222509 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:29.222243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerStarted","Data":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} Jul 09 23:29:29.250374 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:29.250326 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.607725631 podStartE2EDuration="7.250308785s" totalImagesPullingTime="5.624723193s" totalInitContainerRuntime="17.859961ms" isStatefulPod=false podCreationTimestamp="2026-07-09 23:29:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:29:29.247760557 +0000 UTC m=+97.128512542" watchObservedRunningTime="2026-07-09 23:29:29.250308785 +0000 UTC m=+97.131060745" Jul 09 23:29:30.116624 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:30.116589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:30.217017 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:30.216982 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:31.490218 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:31.490160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:32.840646 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:32.840613 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:29:35.923490 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:35.923454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:36.022033 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:36.021994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:36.411611 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:36.411577 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:36.512622 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:36.512587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:37.083994 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:37.083969 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mhsqr" Jul 09 23:29:47.196521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.196448 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-848bc7b456-hwrpb" podUID="20c269e9-7523-475b-ac55-bfd29c5d9944" containerName="console" containerID="cri-o://06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad" gracePeriod=15 Jul 09 23:29:47.478134 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.478055 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:47.499636 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.499607 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-oauth-config\") pod \"20c269e9-7523-475b-ac55-bfd29c5d9944\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " Jul 09 23:29:47.499825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.499643 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-console-config\") pod \"20c269e9-7523-475b-ac55-bfd29c5d9944\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " Jul 09 23:29:47.499825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.499703 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-oauth-serving-cert\") pod \"20c269e9-7523-475b-ac55-bfd29c5d9944\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " Jul 09 23:29:47.499825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.499728 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-service-ca\") pod \"20c269e9-7523-475b-ac55-bfd29c5d9944\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " Jul 09 23:29:47.499825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.499768 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-trusted-ca-bundle\") pod \"20c269e9-7523-475b-ac55-bfd29c5d9944\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " Jul 09 23:29:47.499825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.499801 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwmd7\" (UniqueName: \"kubernetes.io/projected/20c269e9-7523-475b-ac55-bfd29c5d9944-kube-api-access-jwmd7\") pod \"20c269e9-7523-475b-ac55-bfd29c5d9944\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " Jul 09 23:29:47.500946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.499837 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-serving-cert\") pod \"20c269e9-7523-475b-ac55-bfd29c5d9944\" (UID: \"20c269e9-7523-475b-ac55-bfd29c5d9944\") " Jul 09 23:29:47.500946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.500146 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-console-config" (OuterVolumeSpecName: "console-config") pod "20c269e9-7523-475b-ac55-bfd29c5d9944" (UID: "20c269e9-7523-475b-ac55-bfd29c5d9944"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:47.500946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.500242 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-service-ca" (OuterVolumeSpecName: "service-ca") pod "20c269e9-7523-475b-ac55-bfd29c5d9944" (UID: "20c269e9-7523-475b-ac55-bfd29c5d9944"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:47.500946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.500485 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "20c269e9-7523-475b-ac55-bfd29c5d9944" (UID: "20c269e9-7523-475b-ac55-bfd29c5d9944"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:47.500946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.500799 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "20c269e9-7523-475b-ac55-bfd29c5d9944" (UID: "20c269e9-7523-475b-ac55-bfd29c5d9944"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:47.502688 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.502659 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "20c269e9-7523-475b-ac55-bfd29c5d9944" (UID: "20c269e9-7523-475b-ac55-bfd29c5d9944"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:29:47.502800 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.502710 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "20c269e9-7523-475b-ac55-bfd29c5d9944" (UID: "20c269e9-7523-475b-ac55-bfd29c5d9944"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:29:47.502800 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.502774 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c269e9-7523-475b-ac55-bfd29c5d9944-kube-api-access-jwmd7" (OuterVolumeSpecName: "kube-api-access-jwmd7") pod "20c269e9-7523-475b-ac55-bfd29c5d9944" (UID: "20c269e9-7523-475b-ac55-bfd29c5d9944"). InnerVolumeSpecName "kube-api-access-jwmd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:29:47.604812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.604676 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwmd7\" (UniqueName: \"kubernetes.io/projected/20c269e9-7523-475b-ac55-bfd29c5d9944-kube-api-access-jwmd7\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:47.604812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.604723 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:47.604812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.604735 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/20c269e9-7523-475b-ac55-bfd29c5d9944-console-oauth-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:47.604812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.604748 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-console-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:47.604812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.604760 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-oauth-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:47.604812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.604773 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-service-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:47.604812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:47.604784 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c269e9-7523-475b-ac55-bfd29c5d9944-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:48.300401 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.300364 2562 generic.go:356] "Generic (PLEG): container finished" podID="20c269e9-7523-475b-ac55-bfd29c5d9944" containerID="06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad" exitCode=0 Jul 09 23:29:48.300866 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.300416 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc7b456-hwrpb" Jul 09 23:29:48.300866 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.300419 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc7b456-hwrpb" event={"ID":"20c269e9-7523-475b-ac55-bfd29c5d9944","Type":"ContainerDied","Data":"06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad"} Jul 09 23:29:48.300866 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.300485 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc7b456-hwrpb" event={"ID":"20c269e9-7523-475b-ac55-bfd29c5d9944","Type":"ContainerDied","Data":"4ad0cc730a268024eab8c31b9b91cd13bdc762013f438bc8071c91a5327635f1"} Jul 09 23:29:48.300866 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.300536 2562 scope.go:117] "RemoveContainer" containerID="06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad" Jul 09 23:29:48.320319 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.320293 2562 scope.go:117] "RemoveContainer" containerID="06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad" Jul 09 23:29:48.320646 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:48.320619 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad\": container with ID starting with 06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad not found: ID does not exist" containerID="06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad" Jul 09 23:29:48.320732 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.320655 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad"} err="failed to get container status \"06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad\": rpc error: code = NotFound desc = could not find container \"06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad\": container with ID starting with 06b1edf571d7d63778547cf3722ebe7ba40f73def825e8eae1341ec1a98913ad not found: ID does not exist" Jul 09 23:29:48.321131 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.321113 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848bc7b456-hwrpb"] Jul 09 23:29:48.324452 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.324433 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-848bc7b456-hwrpb"] Jul 09 23:29:48.616981 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.616907 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c269e9-7523-475b-ac55-bfd29c5d9944" path="/var/lib/kubelet/pods/20c269e9-7523-475b-ac55-bfd29c5d9944/volumes" Jul 09 23:29:48.856122 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.856086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:48.904873 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:48.904790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:50.815022 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:50.814979 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-567bfcdcdb-mpxxn" podUID="a2c63b46-347f-474f-a071-9c731682a89c" containerName="console" containerID="cri-o://e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0" gracePeriod=15 Jul 09 23:29:51.082560 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.082498 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:51.134255 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134219 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-console-config\") pod \"a2c63b46-347f-474f-a071-9c731682a89c\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " Jul 09 23:29:51.134429 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134263 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-service-ca\") pod \"a2c63b46-347f-474f-a071-9c731682a89c\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " Jul 09 23:29:51.134429 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134302 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-oauth-serving-cert\") pod \"a2c63b46-347f-474f-a071-9c731682a89c\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " Jul 09 23:29:51.134429 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134331 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-serving-cert\") pod \"a2c63b46-347f-474f-a071-9c731682a89c\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " Jul 09 23:29:51.134429 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134379 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-oauth-config\") pod \"a2c63b46-347f-474f-a071-9c731682a89c\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " Jul 09 23:29:51.134429 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134425 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-trusted-ca-bundle\") pod \"a2c63b46-347f-474f-a071-9c731682a89c\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " Jul 09 23:29:51.134761 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134461 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9wvv\" (UniqueName: \"kubernetes.io/projected/a2c63b46-347f-474f-a071-9c731682a89c-kube-api-access-g9wvv\") pod \"a2c63b46-347f-474f-a071-9c731682a89c\" (UID: \"a2c63b46-347f-474f-a071-9c731682a89c\") " Jul 09 23:29:51.134761 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134683 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-console-config" (OuterVolumeSpecName: "console-config") pod "a2c63b46-347f-474f-a071-9c731682a89c" (UID: "a2c63b46-347f-474f-a071-9c731682a89c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:51.134871 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134685 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-service-ca" (OuterVolumeSpecName: "service-ca") pod "a2c63b46-347f-474f-a071-9c731682a89c" (UID: "a2c63b46-347f-474f-a071-9c731682a89c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:51.134871 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134781 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a2c63b46-347f-474f-a071-9c731682a89c" (UID: "a2c63b46-347f-474f-a071-9c731682a89c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:51.134962 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.134882 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a2c63b46-347f-474f-a071-9c731682a89c" (UID: "a2c63b46-347f-474f-a071-9c731682a89c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:29:51.136724 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.136695 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a2c63b46-347f-474f-a071-9c731682a89c" (UID: "a2c63b46-347f-474f-a071-9c731682a89c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:29:51.136850 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.136761 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a2c63b46-347f-474f-a071-9c731682a89c" (UID: "a2c63b46-347f-474f-a071-9c731682a89c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:29:51.136850 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.136772 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2c63b46-347f-474f-a071-9c731682a89c-kube-api-access-g9wvv" (OuterVolumeSpecName: "kube-api-access-g9wvv") pod "a2c63b46-347f-474f-a071-9c731682a89c" (UID: "a2c63b46-347f-474f-a071-9c731682a89c"). InnerVolumeSpecName "kube-api-access-g9wvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:29:51.235574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.235543 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:51.235574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.235570 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9wvv\" (UniqueName: \"kubernetes.io/projected/a2c63b46-347f-474f-a071-9c731682a89c-kube-api-access-g9wvv\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:51.235574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.235582 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-console-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:51.235787 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.235590 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-service-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:51.235787 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.235598 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2c63b46-347f-474f-a071-9c731682a89c-oauth-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:51.235787 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.235607 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-serving-cert\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:51.235787 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.235615 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2c63b46-347f-474f-a071-9c731682a89c-console-oauth-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:29:51.313785 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.313747 2562 generic.go:356] "Generic (PLEG): container finished" podID="a2c63b46-347f-474f-a071-9c731682a89c" containerID="e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0" exitCode=0 Jul 09 23:29:51.313954 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.313795 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-567bfcdcdb-mpxxn" Jul 09 23:29:51.313954 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.313829 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bfcdcdb-mpxxn" event={"ID":"a2c63b46-347f-474f-a071-9c731682a89c","Type":"ContainerDied","Data":"e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0"} Jul 09 23:29:51.313954 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.313883 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-567bfcdcdb-mpxxn" event={"ID":"a2c63b46-347f-474f-a071-9c731682a89c","Type":"ContainerDied","Data":"48d2522fc6648e16e2ade1079a97cc25864d089161bf8a2f9374dab530dee70d"} Jul 09 23:29:51.313954 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.313902 2562 scope.go:117] "RemoveContainer" containerID="e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0" Jul 09 23:29:51.326982 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.326961 2562 scope.go:117] "RemoveContainer" containerID="e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0" Jul 09 23:29:51.327324 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:29:51.327296 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0\": container with ID starting with e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0 not found: ID does not exist" containerID="e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0" Jul 09 23:29:51.327400 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.327337 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0"} err="failed to get container status \"e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0\": rpc error: code = NotFound desc = could not find container \"e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0\": container with ID starting with e9f0e668550868077888e70c298fde5b2e40303b6cdaded693a51b76cb19fbb0 not found: ID does not exist" Jul 09 23:29:51.336538 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.336487 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-567bfcdcdb-mpxxn"] Jul 09 23:29:51.340216 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:51.340177 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-567bfcdcdb-mpxxn"] Jul 09 23:29:52.344573 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:52.344533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:52.618296 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:52.618228 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2c63b46-347f-474f-a071-9c731682a89c" path="/var/lib/kubelet/pods/a2c63b46-347f-474f-a071-9c731682a89c/volumes" Jul 09 23:29:56.182853 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:56.182817 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:56.283825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:56.283788 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:29:56.384556 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:29:56.384521 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:03.846375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:03.846339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:03.946368 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:03.946332 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:20.712564 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:20.712529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:22.841577 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:22.841546 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:22.860828 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:22.860798 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:23.455170 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:23.455140 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:32.407291 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:32.407252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:32.473279 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:32.473244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:33.513571 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:33.513538 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:33.563003 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:33.562974 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:33.618923 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:33.618893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:33.680535 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:33.680507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:34.217518 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:34.217488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:34.266636 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:34.266610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:36.702340 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:36.702300 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:30:36.702723 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:36.702605 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="alertmanager" containerID="cri-o://6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31" gracePeriod=120 Jul 09 23:30:36.702785 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:36.702707 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-web" containerID="cri-o://02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4" gracePeriod=120 Jul 09 23:30:36.702785 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:36.702721 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="prom-label-proxy" containerID="cri-o://86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67" gracePeriod=120 Jul 09 23:30:36.702785 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:36.702751 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy" containerID="cri-o://25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8" gracePeriod=120 Jul 09 23:30:36.702939 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:36.702699 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-metric" containerID="cri-o://b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c" gracePeriod=120 Jul 09 23:30:36.702939 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:36.702780 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="config-reloader" containerID="cri-o://47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3" gracePeriod=120 Jul 09 23:30:37.492501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492469 2562 generic.go:356] "Generic (PLEG): container finished" podID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerID="86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67" exitCode=0 Jul 09 23:30:37.492501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492494 2562 generic.go:356] "Generic (PLEG): container finished" podID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerID="b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c" exitCode=0 Jul 09 23:30:37.492501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492502 2562 generic.go:356] "Generic (PLEG): container finished" podID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerID="25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8" exitCode=0 Jul 09 23:30:37.492501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492509 2562 generic.go:356] "Generic (PLEG): container finished" podID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerID="47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3" exitCode=0 Jul 09 23:30:37.492501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492515 2562 generic.go:356] "Generic (PLEG): container finished" podID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerID="6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31" exitCode=0 Jul 09 23:30:37.492791 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492547 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67"} Jul 09 23:30:37.492791 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492623 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c"} Jul 09 23:30:37.492791 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492649 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8"} Jul 09 23:30:37.492791 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492674 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3"} Jul 09 23:30:37.492791 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.492697 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31"} Jul 09 23:30:37.945451 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:37.945427 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.029459 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029423 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-volume\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029638 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029469 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029638 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029505 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029638 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029610 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-metrics-client-ca\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029815 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029663 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59gc5\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-kube-api-access-59gc5\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029815 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029698 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-out\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029815 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029741 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-web\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029815 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029785 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-tls-assets\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.029815 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029813 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.030073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029848 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-main-db\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.030073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029874 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-cluster-tls-config\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.030073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029883 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:38.030073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029920 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.030073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029970 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-web-config\") pod \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\" (UID: \"6707c747-a7e4-428b-8ba6-c8b91fe7e94d\") " Jul 09 23:30:38.030073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.029988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:38.031234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.030231 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.031234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.030258 2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-metrics-client-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.031234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.030510 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:30:38.032574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.032544 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:38.032574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.032549 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-kube-api-access-59gc5" (OuterVolumeSpecName: "kube-api-access-59gc5") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "kube-api-access-59gc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:30:38.032715 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.032604 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-out" (OuterVolumeSpecName: "config-out") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:30:38.032715 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.032622 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-volume" (OuterVolumeSpecName: "config-volume") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:38.033608 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.033585 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:38.033976 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.033939 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:38.034628 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.034609 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:38.034763 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.034746 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:30:38.038148 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.038118 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:38.044735 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.044705 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-web-config" (OuterVolumeSpecName: "web-config") pod "6707c747-a7e4-428b-8ba6-c8b91fe7e94d" (UID: "6707c747-a7e4-428b-8ba6-c8b91fe7e94d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131414 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131442 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-tls-assets\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131454 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131465 2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-alertmanager-main-db\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131474 2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-cluster-tls-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131482 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131491 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-web-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131499 2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-volume\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131827 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131507 2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-secret-alertmanager-main-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131827 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131516 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59gc5\" (UniqueName: \"kubernetes.io/projected/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-kube-api-access-59gc5\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.131827 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.131524 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6707c747-a7e4-428b-8ba6-c8b91fe7e94d-config-out\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:38.500100 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.500012 2562 generic.go:356] "Generic (PLEG): container finished" podID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerID="02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4" exitCode=0 Jul 09 23:30:38.500100 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.500093 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.500290 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.500098 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4"} Jul 09 23:30:38.500290 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.500186 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6707c747-a7e4-428b-8ba6-c8b91fe7e94d","Type":"ContainerDied","Data":"6b4c810dfd6b7502e8ef570a7cb1fe2c26d7589780730d837b59030b71672964"} Jul 09 23:30:38.500290 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.500228 2562 scope.go:117] "RemoveContainer" containerID="86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67" Jul 09 23:30:38.507959 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.507946 2562 scope.go:117] "RemoveContainer" containerID="b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c" Jul 09 23:30:38.515156 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.515142 2562 scope.go:117] "RemoveContainer" containerID="25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8" Jul 09 23:30:38.522720 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.522698 2562 scope.go:117] "RemoveContainer" containerID="02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4" Jul 09 23:30:38.522991 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.522970 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:30:38.530707 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.527716 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:30:38.534747 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.534730 2562 scope.go:117] "RemoveContainer" containerID="47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3" Jul 09 23:30:38.541684 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.541667 2562 scope.go:117] "RemoveContainer" containerID="6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31" Jul 09 23:30:38.549043 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.549026 2562 scope.go:117] "RemoveContainer" containerID="d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808" Jul 09 23:30:38.551512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551493 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:30:38.551778 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551764 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-web" Jul 09 23:30:38.551778 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551777 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-web" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551786 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c269e9-7523-475b-ac55-bfd29c5d9944" containerName="console" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551792 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c269e9-7523-475b-ac55-bfd29c5d9944" containerName="console" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551799 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="config-reloader" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551804 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="config-reloader" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551815 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2c63b46-347f-474f-a071-9c731682a89c" containerName="console" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551820 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2c63b46-347f-474f-a071-9c731682a89c" containerName="console" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551825 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-metric" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551830 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-metric" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551835 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="prom-label-proxy" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551839 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="prom-label-proxy" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551847 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="init-config-reloader" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551852 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="init-config-reloader" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551863 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551868 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551880 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="alertmanager" Jul 09 23:30:38.551901 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551884 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="alertmanager" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551930 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-web" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551939 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551944 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="config-reloader" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551951 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c269e9-7523-475b-ac55-bfd29c5d9944" containerName="console" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551956 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2c63b46-347f-474f-a071-9c731682a89c" containerName="console" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551960 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="prom-label-proxy" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551967 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="kube-rbac-proxy-metric" Jul 09 23:30:38.552704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.551973 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" containerName="alertmanager" Jul 09 23:30:38.557322 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.557150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.557322 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.557294 2562 scope.go:117] "RemoveContainer" containerID="86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67" Jul 09 23:30:38.557592 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:38.557567 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67\": container with ID starting with 86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67 not found: ID does not exist" containerID="86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67" Jul 09 23:30:38.557681 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.557600 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67"} err="failed to get container status \"86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67\": rpc error: code = NotFound desc = could not find container \"86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67\": container with ID starting with 86a31397d9c8e5354c787e96030bf96864234ba24c1f5e6217f5c5959f1fbd67 not found: ID does not exist" Jul 09 23:30:38.557681 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.557625 2562 scope.go:117] "RemoveContainer" containerID="b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c" Jul 09 23:30:38.557897 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:38.557878 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c\": container with ID starting with b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c not found: ID does not exist" containerID="b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c" Jul 09 23:30:38.557967 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.557907 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c"} err="failed to get container status \"b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c\": rpc error: code = NotFound desc = could not find container \"b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c\": container with ID starting with b527c51bbfd8ee1ee43d43aea757b80aae28c8172794708f64d54946244ec41c not found: ID does not exist" Jul 09 23:30:38.557967 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.557930 2562 scope.go:117] "RemoveContainer" containerID="25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8" Jul 09 23:30:38.558216 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:38.558183 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8\": container with ID starting with 25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8 not found: ID does not exist" containerID="25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8" Jul 09 23:30:38.558271 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.558222 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8"} err="failed to get container status \"25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8\": rpc error: code = NotFound desc = could not find container \"25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8\": container with ID starting with 25669ea428b7e32123423de77e70518b99577c5d6a732341a6d3a0e1d0b53ce8 not found: ID does not exist" Jul 09 23:30:38.558271 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.558238 2562 scope.go:117] "RemoveContainer" containerID="02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4" Jul 09 23:30:38.558477 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:38.558461 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4\": container with ID starting with 02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4 not found: ID does not exist" containerID="02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4" Jul 09 23:30:38.558537 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.558485 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4"} err="failed to get container status \"02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4\": rpc error: code = NotFound desc = could not find container \"02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4\": container with ID starting with 02f76a4eeaf0abc765fb15b8bf6ddfb17f6c72cc0255f54db32eaaf9f0b9b4b4 not found: ID does not exist" Jul 09 23:30:38.558537 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.558505 2562 scope.go:117] "RemoveContainer" containerID="47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3" Jul 09 23:30:38.558758 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:38.558733 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3\": container with ID starting with 47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3 not found: ID does not exist" containerID="47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3" Jul 09 23:30:38.558843 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.558765 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3"} err="failed to get container status \"47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3\": rpc error: code = NotFound desc = could not find container \"47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3\": container with ID starting with 47db18167b86d5bdf5e1e29de58d56fc5e375aeabc94b9b8a69511c751a9c0b3 not found: ID does not exist" Jul 09 23:30:38.558843 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.558784 2562 scope.go:117] "RemoveContainer" containerID="6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31" Jul 09 23:30:38.559057 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:38.559033 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31\": container with ID starting with 6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31 not found: ID does not exist" containerID="6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31" Jul 09 23:30:38.559124 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.559060 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31"} err="failed to get container status \"6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31\": rpc error: code = NotFound desc = could not find container \"6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31\": container with ID starting with 6b23e0acee315f8941bfb7e994faa087c373292a10d18d3dcef0845a5557bf31 not found: ID does not exist" Jul 09 23:30:38.559124 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.559075 2562 scope.go:117] "RemoveContainer" containerID="d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808" Jul 09 23:30:38.559382 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:38.559361 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808\": container with ID starting with d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808 not found: ID does not exist" containerID="d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808" Jul 09 23:30:38.559449 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.559385 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808"} err="failed to get container status \"d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808\": rpc error: code = NotFound desc = could not find container \"d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808\": container with ID starting with d8dd4e2ab1fa276880794584810b682d45af8c24c02f7785ff46971e038c5808 not found: ID does not exist" Jul 09 23:30:38.561375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.560147 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 09 23:30:38.561375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.560541 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 09 23:30:38.561375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.560969 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 09 23:30:38.561375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.561188 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 09 23:30:38.561684 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.561419 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 09 23:30:38.561684 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.561454 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 09 23:30:38.561684 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.561532 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 09 23:30:38.561684 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.561647 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 09 23:30:38.561923 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.561898 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pqnr6\"" Jul 09 23:30:38.564911 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.564890 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 09 23:30:38.570151 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.570129 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:30:38.616532 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.616503 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6707c747-a7e4-428b-8ba6-c8b91fe7e94d" path="/var/lib/kubelet/pods/6707c747-a7e4-428b-8ba6-c8b91fe7e94d/volumes" Jul 09 23:30:38.634533 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634703 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634703 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634703 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-config-out\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634703 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634689 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dsgc\" (UniqueName: \"kubernetes.io/projected/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-kube-api-access-2dsgc\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634899 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634899 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-web-config\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634899 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634899 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634813 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.634899 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634874 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.635126 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.635126 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.635126 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.634967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735531 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735705 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-config-out\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735705 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dsgc\" (UniqueName: \"kubernetes.io/projected/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-kube-api-access-2dsgc\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735705 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735705 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-web-config\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735705 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735705 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.735705 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.736049 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.735940 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.736104 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.736060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.736155 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.736098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.736155 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.736137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.736287 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.736167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.736287 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.736237 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.736477 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.736454 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.738034 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.738011 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.738982 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.738955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.739078 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.739026 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-config-out\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.739078 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.739044 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.739315 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.739297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-config-volume\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.739381 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.739365 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.739512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.739491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.739581 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.739535 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-web-config\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.740573 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.740554 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.740682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.740663 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.742825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.742806 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dsgc\" (UniqueName: \"kubernetes.io/projected/4d8cffea-f5b7-47dc-931f-9b7acfbe1b32-kube-api-access-2dsgc\") pod \"alertmanager-main-0\" (UID: \"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32\") " pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:38.868786 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:38.868749 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 09 23:30:39.005361 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:39.005331 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 09 23:30:39.008273 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:30:39.008243 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8cffea_f5b7_47dc_931f_9b7acfbe1b32.slice/crio-cbee3cf7fd3d0032a04fec17f426a1312625a9aded886c783be31bf34b88eea9 WatchSource:0}: Error finding container cbee3cf7fd3d0032a04fec17f426a1312625a9aded886c783be31bf34b88eea9: Status 404 returned error can't find the container with id cbee3cf7fd3d0032a04fec17f426a1312625a9aded886c783be31bf34b88eea9 Jul 09 23:30:39.506619 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:39.506590 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d8cffea-f5b7-47dc-931f-9b7acfbe1b32" containerID="f8db79e20b41d5333966ed38d2603b43a5f9bab9dc73f832c8158b05407cfab2" exitCode=0 Jul 09 23:30:39.506743 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:39.506666 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerDied","Data":"f8db79e20b41d5333966ed38d2603b43a5f9bab9dc73f832c8158b05407cfab2"} Jul 09 23:30:39.506743 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:39.506732 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerStarted","Data":"cbee3cf7fd3d0032a04fec17f426a1312625a9aded886c783be31bf34b88eea9"} Jul 09 23:30:40.513365 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.513329 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerStarted","Data":"3e897c22f78b78899d24ef36839a8bbe29b02da8f8a404da37da1e562501dad3"} Jul 09 23:30:40.513729 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.513401 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerStarted","Data":"59de0d4ed9c0d355240e0ac513c761915a63884326330db4cfa25e84a5124e7a"} Jul 09 23:30:40.513729 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.513441 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerStarted","Data":"90264f7412a6300192ebb26c05edf2738c41d40ab48725327b39c487929ddc9f"} Jul 09 23:30:40.513729 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.513470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerStarted","Data":"43ef2fea925c3b02ebedb2a4c544fe5a82d96f6e65d183bf2f7b3a4a53d93598"} Jul 09 23:30:40.513729 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.513503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerStarted","Data":"5309c757d30f9fa8f9f38c8b151b497e8b0bb584a2743f937b5d6ac618f2398f"} Jul 09 23:30:40.513729 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.513541 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4d8cffea-f5b7-47dc-931f-9b7acfbe1b32","Type":"ContainerStarted","Data":"0a40d6dccfe2f3daea250b685e5bf2ee5b4bd7bad060c80f6cad592bc0a120d3"} Jul 09 23:30:40.539533 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.539468 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.524317698 podStartE2EDuration="2.539451706s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.134008ms" isStatefulPod=false podCreationTimestamp="2026-07-09 23:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:30:40.538033987 +0000 UTC m=+168.418785947" watchObservedRunningTime="2026-07-09 23:30:40.539451706 +0000 UTC m=+168.420203666" Jul 09 23:30:40.927407 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.927320 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:30:40.928135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.927670 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="prometheus" containerID="cri-o://62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" gracePeriod=600 Jul 09 23:30:40.928135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.927849 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-web" containerID="cri-o://01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" gracePeriod=600 Jul 09 23:30:40.928135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.927850 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy" containerID="cri-o://d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" gracePeriod=600 Jul 09 23:30:40.928135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.927930 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="thanos-sidecar" containerID="cri-o://644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" gracePeriod=600 Jul 09 23:30:40.928135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.928000 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="config-reloader" containerID="cri-o://61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" gracePeriod=600 Jul 09 23:30:40.928135 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:40.928012 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" gracePeriod=600 Jul 09 23:30:41.190580 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.190548 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.257337 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257308 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-metrics-client-ca\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257337 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257339 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-trusted-ca-bundle\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257356 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-0\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257382 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-thanos-prometheus-http-client-file\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257412 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-tls\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257441 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-serving-certs-ca-bundle\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257468 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257507 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-tls-assets\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257535 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257555 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-config\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257577 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cflp6\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-kube-api-access-cflp6\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257608 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-1\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257626 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-web-config\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257641 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-kube-rbac-proxy\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257659 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-2\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257683 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-config-out\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257725 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-db\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257719 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257758 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-metrics-client-certs\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257786 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-kubelet-serving-ca-bundle\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257817 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-grpc-tls\") pod \"0016e619-ebdd-46de-92e6-281d4a75071c\" (UID: \"0016e619-ebdd-46de-92e6-281d4a75071c\") " Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257862 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:41.257971 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.257871 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:41.258599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.258113 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-metrics-client-ca\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.258599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.258135 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.258599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.258156 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.258599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.258555 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:41.259035 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.258984 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:41.260819 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.259657 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:30:41.260819 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.259670 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:41.260819 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.260442 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:30:41.261687 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.261658 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.262171 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.262044 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-config-out" (OuterVolumeSpecName: "config-out") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:30:41.263189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263140 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-kube-api-access-cflp6" (OuterVolumeSpecName: "kube-api-access-cflp6") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "kube-api-access-cflp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:30:41.263189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263158 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:30:41.263456 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263403 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.263624 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263530 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.263624 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263600 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.263739 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263627 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.263739 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263643 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.263739 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.263723 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.264707 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.264689 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-config" (OuterVolumeSpecName: "config") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.273454 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.273427 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-web-config" (OuterVolumeSpecName: "web-config") pod "0016e619-ebdd-46de-92e6-281d4a75071c" (UID: "0016e619-ebdd-46de-92e6-281d4a75071c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:30:41.358992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.358955 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.358992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.358984 2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-config-out\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.358992 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.358995 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-db\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359007 2562 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-metrics-client-certs\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359018 2562 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359027 2562 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-grpc-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359035 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359044 2562 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359052 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359060 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359070 2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-tls-assets\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359078 2562 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359087 2562 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359095 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cflp6\" (UniqueName: \"kubernetes.io/projected/0016e619-ebdd-46de-92e6-281d4a75071c-kube-api-access-cflp6\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359102 2562 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0016e619-ebdd-46de-92e6-281d4a75071c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359131 2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-web-config\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.359250 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.359140 2562 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0016e619-ebdd-46de-92e6-281d4a75071c-secret-kube-rbac-proxy\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:30:41.519693 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519660 2562 generic.go:356] "Generic (PLEG): container finished" podID="0016e619-ebdd-46de-92e6-281d4a75071c" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" exitCode=0 Jul 09 23:30:41.519693 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519689 2562 generic.go:356] "Generic (PLEG): container finished" podID="0016e619-ebdd-46de-92e6-281d4a75071c" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" exitCode=0 Jul 09 23:30:41.519693 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519700 2562 generic.go:356] "Generic (PLEG): container finished" podID="0016e619-ebdd-46de-92e6-281d4a75071c" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" exitCode=0 Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519705 2562 generic.go:356] "Generic (PLEG): container finished" podID="0016e619-ebdd-46de-92e6-281d4a75071c" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" exitCode=0 Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519711 2562 generic.go:356] "Generic (PLEG): container finished" podID="0016e619-ebdd-46de-92e6-281d4a75071c" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" exitCode=0 Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519716 2562 generic.go:356] "Generic (PLEG): container finished" podID="0016e619-ebdd-46de-92e6-281d4a75071c" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" exitCode=0 Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519734 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519879 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519928 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519952 2562 scope.go:117] "RemoveContainer" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.519972 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.520013 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.520050 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} Jul 09 23:30:41.520300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.520085 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0016e619-ebdd-46de-92e6-281d4a75071c","Type":"ContainerDied","Data":"daaabc41d0ef79274b72e5b8f05fa9726b10ee5e317267005cfd05f1b5d75413"} Jul 09 23:30:41.528457 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.528438 2562 scope.go:117] "RemoveContainer" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.535357 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.535339 2562 scope.go:117] "RemoveContainer" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.542380 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.542358 2562 scope.go:117] "RemoveContainer" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.544144 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.544122 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:30:41.549933 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.549892 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:30:41.550143 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.550128 2562 scope.go:117] "RemoveContainer" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.557018 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.557001 2562 scope.go:117] "RemoveContainer" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.564438 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.564411 2562 scope.go:117] "RemoveContainer" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.571330 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.571315 2562 scope.go:117] "RemoveContainer" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.571582 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:41.571564 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": container with ID starting with d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0 not found: ID does not exist" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.571630 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.571594 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} err="failed to get container status \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": rpc error: code = NotFound desc = could not find container \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": container with ID starting with d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0 not found: ID does not exist" Jul 09 23:30:41.571630 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.571614 2562 scope.go:117] "RemoveContainer" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.571829 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:41.571812 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": container with ID starting with d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0 not found: ID does not exist" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.571864 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.571836 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} err="failed to get container status \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": rpc error: code = NotFound desc = could not find container \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": container with ID starting with d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0 not found: ID does not exist" Jul 09 23:30:41.571864 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.571852 2562 scope.go:117] "RemoveContainer" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.572045 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:41.572027 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": container with ID starting with 01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7 not found: ID does not exist" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.572100 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.572048 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} err="failed to get container status \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": rpc error: code = NotFound desc = could not find container \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": container with ID starting with 01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7 not found: ID does not exist" Jul 09 23:30:41.572100 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.572070 2562 scope.go:117] "RemoveContainer" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.572339 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:41.572301 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": container with ID starting with 644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e not found: ID does not exist" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.572506 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.572332 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} err="failed to get container status \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": rpc error: code = NotFound desc = could not find container \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": container with ID starting with 644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e not found: ID does not exist" Jul 09 23:30:41.572506 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.572351 2562 scope.go:117] "RemoveContainer" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.572758 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:41.572734 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": container with ID starting with 61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47 not found: ID does not exist" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.572825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.572768 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} err="failed to get container status \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": rpc error: code = NotFound desc = could not find container \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": container with ID starting with 61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47 not found: ID does not exist" Jul 09 23:30:41.572825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.572805 2562 scope.go:117] "RemoveContainer" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.573253 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:41.573234 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": container with ID starting with 62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31 not found: ID does not exist" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.573355 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.573261 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} err="failed to get container status \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": rpc error: code = NotFound desc = could not find container \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": container with ID starting with 62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31 not found: ID does not exist" Jul 09 23:30:41.573355 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.573280 2562 scope.go:117] "RemoveContainer" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.573571 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:30:41.573553 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": container with ID starting with 4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92 not found: ID does not exist" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.573617 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.573584 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92"} err="failed to get container status \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": rpc error: code = NotFound desc = could not find container \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": container with ID starting with 4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92 not found: ID does not exist" Jul 09 23:30:41.573617 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.573598 2562 scope.go:117] "RemoveContainer" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.573854 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.573836 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} err="failed to get container status \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": rpc error: code = NotFound desc = could not find container \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": container with ID starting with d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0 not found: ID does not exist" Jul 09 23:30:41.573912 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.573855 2562 scope.go:117] "RemoveContainer" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.574126 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574108 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} err="failed to get container status \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": rpc error: code = NotFound desc = could not find container \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": container with ID starting with d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0 not found: ID does not exist" Jul 09 23:30:41.574195 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574128 2562 scope.go:117] "RemoveContainer" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.574346 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574328 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:30:41.574395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574383 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} err="failed to get container status \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": rpc error: code = NotFound desc = could not find container \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": container with ID starting with 01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7 not found: ID does not exist" Jul 09 23:30:41.574440 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574397 2562 scope.go:117] "RemoveContainer" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.574625 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574607 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} err="failed to get container status \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": rpc error: code = NotFound desc = could not find container \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": container with ID starting with 644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e not found: ID does not exist" Jul 09 23:30:41.574625 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574620 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="config-reloader" Jul 09 23:30:41.574744 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574627 2562 scope.go:117] "RemoveContainer" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.574744 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574631 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="config-reloader" Jul 09 23:30:41.574744 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574712 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="thanos-sidecar" Jul 09 23:30:41.574744 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574727 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="thanos-sidecar" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574767 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="init-config-reloader" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574778 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="init-config-reloader" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574788 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-web" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574798 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-web" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574810 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="prometheus" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574819 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="prometheus" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574835 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-thanos" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574844 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-thanos" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574854 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574862 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574863 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} err="failed to get container status \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": rpc error: code = NotFound desc = could not find container \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": container with ID starting with 61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47 not found: ID does not exist" Jul 09 23:30:41.574935 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574879 2562 scope.go:117] "RemoveContainer" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574944 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-web" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574955 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy-thanos" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574966 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="prometheus" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574974 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="config-reloader" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574983 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="thanos-sidecar" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.574992 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" containerName="kube-rbac-proxy" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575082 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} err="failed to get container status \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": rpc error: code = NotFound desc = could not find container \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": container with ID starting with 62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31 not found: ID does not exist" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575096 2562 scope.go:117] "RemoveContainer" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575285 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92"} err="failed to get container status \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": rpc error: code = NotFound desc = could not find container \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": container with ID starting with 4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92 not found: ID does not exist" Jul 09 23:30:41.575495 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575302 2562 scope.go:117] "RemoveContainer" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.575946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575504 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} err="failed to get container status \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": rpc error: code = NotFound desc = could not find container \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": container with ID starting with d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0 not found: ID does not exist" Jul 09 23:30:41.575946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575519 2562 scope.go:117] "RemoveContainer" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.575946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575690 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} err="failed to get container status \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": rpc error: code = NotFound desc = could not find container \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": container with ID starting with d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0 not found: ID does not exist" Jul 09 23:30:41.575946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575704 2562 scope.go:117] "RemoveContainer" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.575946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575865 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} err="failed to get container status \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": rpc error: code = NotFound desc = could not find container \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": container with ID starting with 01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7 not found: ID does not exist" Jul 09 23:30:41.575946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.575879 2562 scope.go:117] "RemoveContainer" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.576245 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.576068 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} err="failed to get container status \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": rpc error: code = NotFound desc = could not find container \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": container with ID starting with 644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e not found: ID does not exist" Jul 09 23:30:41.576245 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.576080 2562 scope.go:117] "RemoveContainer" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.576416 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.576388 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} err="failed to get container status \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": rpc error: code = NotFound desc = could not find container \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": container with ID starting with 61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47 not found: ID does not exist" Jul 09 23:30:41.576519 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.576423 2562 scope.go:117] "RemoveContainer" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.576758 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.576735 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} err="failed to get container status \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": rpc error: code = NotFound desc = could not find container \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": container with ID starting with 62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31 not found: ID does not exist" Jul 09 23:30:41.576861 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.576759 2562 scope.go:117] "RemoveContainer" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.577005 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.576988 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92"} err="failed to get container status \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": rpc error: code = NotFound desc = could not find container \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": container with ID starting with 4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92 not found: ID does not exist" Jul 09 23:30:41.577083 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577006 2562 scope.go:117] "RemoveContainer" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.577261 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577240 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} err="failed to get container status \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": rpc error: code = NotFound desc = could not find container \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": container with ID starting with d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0 not found: ID does not exist" Jul 09 23:30:41.577317 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577262 2562 scope.go:117] "RemoveContainer" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.577489 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577475 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} err="failed to get container status \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": rpc error: code = NotFound desc = could not find container \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": container with ID starting with d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0 not found: ID does not exist" Jul 09 23:30:41.577489 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577488 2562 scope.go:117] "RemoveContainer" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.577728 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577712 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} err="failed to get container status \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": rpc error: code = NotFound desc = could not find container \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": container with ID starting with 01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7 not found: ID does not exist" Jul 09 23:30:41.577767 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577728 2562 scope.go:117] "RemoveContainer" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.577956 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577941 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} err="failed to get container status \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": rpc error: code = NotFound desc = could not find container \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": container with ID starting with 644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e not found: ID does not exist" Jul 09 23:30:41.578002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.577956 2562 scope.go:117] "RemoveContainer" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.578153 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578138 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} err="failed to get container status \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": rpc error: code = NotFound desc = could not find container \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": container with ID starting with 61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47 not found: ID does not exist" Jul 09 23:30:41.578211 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578153 2562 scope.go:117] "RemoveContainer" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.578382 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578364 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} err="failed to get container status \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": rpc error: code = NotFound desc = could not find container \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": container with ID starting with 62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31 not found: ID does not exist" Jul 09 23:30:41.578444 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578384 2562 scope.go:117] "RemoveContainer" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.578624 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578607 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92"} err="failed to get container status \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": rpc error: code = NotFound desc = could not find container \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": container with ID starting with 4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92 not found: ID does not exist" Jul 09 23:30:41.578694 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578625 2562 scope.go:117] "RemoveContainer" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.578805 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578789 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} err="failed to get container status \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": rpc error: code = NotFound desc = could not find container \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": container with ID starting with d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0 not found: ID does not exist" Jul 09 23:30:41.578869 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578806 2562 scope.go:117] "RemoveContainer" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.579002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.578981 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} err="failed to get container status \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": rpc error: code = NotFound desc = could not find container \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": container with ID starting with d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0 not found: ID does not exist" Jul 09 23:30:41.579052 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579004 2562 scope.go:117] "RemoveContainer" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.579231 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579215 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} err="failed to get container status \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": rpc error: code = NotFound desc = could not find container \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": container with ID starting with 01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7 not found: ID does not exist" Jul 09 23:30:41.579280 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579232 2562 scope.go:117] "RemoveContainer" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.579425 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579409 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} err="failed to get container status \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": rpc error: code = NotFound desc = could not find container \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": container with ID starting with 644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e not found: ID does not exist" Jul 09 23:30:41.579493 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579427 2562 scope.go:117] "RemoveContainer" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.579602 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579586 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} err="failed to get container status \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": rpc error: code = NotFound desc = could not find container \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": container with ID starting with 61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47 not found: ID does not exist" Jul 09 23:30:41.579652 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579602 2562 scope.go:117] "RemoveContainer" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.579823 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579807 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} err="failed to get container status \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": rpc error: code = NotFound desc = could not find container \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": container with ID starting with 62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31 not found: ID does not exist" Jul 09 23:30:41.579884 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.579823 2562 scope.go:117] "RemoveContainer" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.580163 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.580060 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92"} err="failed to get container status \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": rpc error: code = NotFound desc = could not find container \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": container with ID starting with 4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92 not found: ID does not exist" Jul 09 23:30:41.580163 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.580082 2562 scope.go:117] "RemoveContainer" containerID="d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0" Jul 09 23:30:41.580723 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.580705 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.581799 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.581762 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0"} err="failed to get container status \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": rpc error: code = NotFound desc = could not find container \"d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0\": container with ID starting with d7d79cf4a09790889833aa9b6161ef7a84afe37075668a79a834d77a86a776d0 not found: ID does not exist" Jul 09 23:30:41.581799 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.581788 2562 scope.go:117] "RemoveContainer" containerID="d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0" Jul 09 23:30:41.582097 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582058 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0"} err="failed to get container status \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": rpc error: code = NotFound desc = could not find container \"d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0\": container with ID starting with d587282f93a0976c0f4df75bbd4eaf3daa33b4ab0fe681de562fe66d331f2db0 not found: ID does not exist" Jul 09 23:30:41.582097 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582088 2562 scope.go:117] "RemoveContainer" containerID="01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7" Jul 09 23:30:41.582388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582364 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7"} err="failed to get container status \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": rpc error: code = NotFound desc = could not find container \"01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7\": container with ID starting with 01486ccb65d2740795df3bd969230a15fae542141fd322a229dcc9f88fdfc0e7 not found: ID does not exist" Jul 09 23:30:41.582446 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582390 2562 scope.go:117] "RemoveContainer" containerID="644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e" Jul 09 23:30:41.582702 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582680 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e"} err="failed to get container status \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": rpc error: code = NotFound desc = could not find container \"644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e\": container with ID starting with 644958fdee65824fcabe64fc387fcfa18254c89f3ef03d2ec709628a1ed4c58e not found: ID does not exist" Jul 09 23:30:41.582772 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582704 2562 scope.go:117] "RemoveContainer" containerID="61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47" Jul 09 23:30:41.582961 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582942 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47"} err="failed to get container status \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": rpc error: code = NotFound desc = could not find container \"61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47\": container with ID starting with 61c7b584115aa8abd2d3abb5c329c6d0d2a2c98d776ca2c46d29ff2ca8308e47 not found: ID does not exist" Jul 09 23:30:41.583042 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.582962 2562 scope.go:117] "RemoveContainer" containerID="62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31" Jul 09 23:30:41.583184 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.583168 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31"} err="failed to get container status \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": rpc error: code = NotFound desc = could not find container \"62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31\": container with ID starting with 62e71bf1f40ea78efc304e6a4a0ed8f9791558180bb07d0a300be474576d8c31 not found: ID does not exist" Jul 09 23:30:41.583263 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.583185 2562 scope.go:117] "RemoveContainer" containerID="4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92" Jul 09 23:30:41.583435 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.583417 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92"} err="failed to get container status \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": rpc error: code = NotFound desc = could not find container \"4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92\": container with ID starting with 4967ff66cb12fd1045f94691aeee57aab1087167d5014319dc9e77a2c7bbaf92 not found: ID does not exist" Jul 09 23:30:41.583890 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.583877 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 23:30:41.583948 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.583876 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 23:30:41.584155 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.584142 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 23:30:41.584496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.584477 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 23:30:41.584618 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.584511 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 23:30:41.584618 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.584568 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pt44r\"" Jul 09 23:30:41.584618 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.584511 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 23:30:41.584811 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.584799 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 23:30:41.585395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.585171 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 23:30:41.585395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.585193 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 23:30:41.585395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.585171 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 23:30:41.585395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.585233 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 23:30:41.585395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.585173 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 23:30:41.585395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.585173 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 23:30:41.585395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.585303 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9o6nu3iaunbns\"" Jul 09 23:30:41.588501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.588469 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 23:30:41.592449 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.592429 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 23:30:41.593496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.593475 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:30:41.661931 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.661888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-config\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.661931 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.661930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662142 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.661994 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662142 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662028 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662142 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662142 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662295 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662150 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662295 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662185 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662295 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662252 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662295 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662422 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662303 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662422 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662336 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3bf99cc-4504-49db-ab8b-30ced52b715a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662480 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3bf99cc-4504-49db-ab8b-30ced52b715a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662510 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662496 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662659 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662717 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.662761 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.662731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7wr\" (UniqueName: \"kubernetes.io/projected/f3bf99cc-4504-49db-ab8b-30ced52b715a-kube-api-access-vj7wr\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763558 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763558 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3bf99cc-4504-49db-ab8b-30ced52b715a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7wr\" (UniqueName: \"kubernetes.io/projected/f3bf99cc-4504-49db-ab8b-30ced52b715a-kube-api-access-vj7wr\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763755 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-config\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.763790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763968 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.763991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.764014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.764046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3bf99cc-4504-49db-ab8b-30ced52b715a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764697 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.764311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764697 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.764533 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764697 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.764547 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764697 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.764597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.764889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.764717 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.765589 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.765558 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.767081 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.766759 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3bf99cc-4504-49db-ab8b-30ced52b715a-config-out\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.767081 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.766854 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.767081 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.766944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3bf99cc-4504-49db-ab8b-30ced52b715a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.767564 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.767501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.767564 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.767534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-web-config\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.767822 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.767727 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.768361 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.768333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.769157 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.769111 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.769373 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.769354 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.769466 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.769451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-config\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.769668 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.769642 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.769802 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.769760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f3bf99cc-4504-49db-ab8b-30ced52b715a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.769861 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.769844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f3bf99cc-4504-49db-ab8b-30ced52b715a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.772293 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.772274 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7wr\" (UniqueName: \"kubernetes.io/projected/f3bf99cc-4504-49db-ab8b-30ced52b715a-kube-api-access-vj7wr\") pod \"prometheus-k8s-0\" (UID: \"f3bf99cc-4504-49db-ab8b-30ced52b715a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:41.893281 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:41.893239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:42.034598 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:42.034516 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 23:30:42.531388 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:42.531348 2562 generic.go:356] "Generic (PLEG): container finished" podID="f3bf99cc-4504-49db-ab8b-30ced52b715a" containerID="53a1387668d7cf2eab1d8c121f456d6b0e8d3c13db764d0381a4ced5dd0f59c3" exitCode=0 Jul 09 23:30:42.531825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:42.531384 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerDied","Data":"53a1387668d7cf2eab1d8c121f456d6b0e8d3c13db764d0381a4ced5dd0f59c3"} Jul 09 23:30:42.531825 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:42.531483 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerStarted","Data":"927c17b2f6b40d8f2d9c343672e3cf1aeac2c3e72cc8e23811dd1629efed08ab"} Jul 09 23:30:42.619359 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:42.619328 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0016e619-ebdd-46de-92e6-281d4a75071c" path="/var/lib/kubelet/pods/0016e619-ebdd-46de-92e6-281d4a75071c/volumes" Jul 09 23:30:43.539395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:43.539319 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerStarted","Data":"7c2b62598b23323a37aea79856af96034c32764d0535c1a8a7c421c8c51cb7b1"} Jul 09 23:30:43.539395 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:43.539386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerStarted","Data":"da385828f58bcf497f47e1802cab1fae184a89eee5a2ce24db19e232f6730ea2"} Jul 09 23:30:43.539781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:43.539411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerStarted","Data":"991e231d926ee12621c168c41851304fc93776c975caab2d4c99fbab492eb11e"} Jul 09 23:30:43.539781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:43.539454 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerStarted","Data":"a1c09d03100ad156205b4887e699fe0fbfaa39fb7403ac7a95ef2b4e5d4ec423"} Jul 09 23:30:43.539781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:43.539483 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerStarted","Data":"95a3e522a5e9db51b60f6ec397a348e747c39ea9f92637bf4b4d4e362c7ae40f"} Jul 09 23:30:43.539781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:43.539507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f3bf99cc-4504-49db-ab8b-30ced52b715a","Type":"ContainerStarted","Data":"647ca72fbe484c2a85fb4f3b8530a77255fefb8e6553b1b149e9eadf90cfdae6"} Jul 09 23:30:43.568405 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:43.568027 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.551194232 podStartE2EDuration="2.568005562s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.81133ms" isStatefulPod=false podCreationTimestamp="2026-07-09 23:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:30:43.564760792 +0000 UTC m=+171.445512754" watchObservedRunningTime="2026-07-09 23:30:43.568005562 +0000 UTC m=+171.448757524" Jul 09 23:30:44.377014 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:44.376979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:44.425436 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:44.425402 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:44.684362 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:44.684266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:44.784738 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:44.784709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:44.830358 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:44.830318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:46.893745 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:46.893707 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:30:48.855076 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:48.855040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:30:48.954812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:30:48.954783 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:23.296895 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:23.296861 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:23.420467 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:23.420426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:32.211865 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:32.211825 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:41.894422 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:41.894368 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:31:41.910889 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:41.910861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:31:42.764147 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:42.764121 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 23:31:49.669350 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:49.669319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:51.035667 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:51.035624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:53.454862 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:53.454832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:53.721174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:53.721088 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:54.275876 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:54.275841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:54.401949 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:54.401908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:55.407357 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:55.407321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:55.508276 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:55.508194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:57.421945 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:57.421912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:31:57.810646 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:31:57.810588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:00.073479 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:00.073443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:00.173521 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:00.173488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:00.273744 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:00.273719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:06.248791 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:06.248752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:08.430560 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:08.430526 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:08.529919 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:08.529882 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:17.341115 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:17.341032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:23.509726 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:23.509690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:23.597517 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:23.597477 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:29.115512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:29.115473 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:29.215176 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:29.215144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:31.222002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:31.221970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:39.449796 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.449759 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:39.490276 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.490240 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw"] Jul 09 23:32:39.493488 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.493470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.497240 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.497218 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2z2tk\"" Jul 09 23:32:39.497366 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.497222 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 23:32:39.497366 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.497360 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 23:32:39.497515 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.497492 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 23:32:39.497623 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.497588 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 23:32:39.497682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.497658 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 23:32:39.502151 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.502123 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw"] Jul 09 23:32:39.529340 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.529306 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:39.560691 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.560654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acbbd40-8570-436c-9ec9-8e21403d663a-cert\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.560822 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.560704 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nff2\" (UniqueName: \"kubernetes.io/projected/2acbbd40-8570-436c-9ec9-8e21403d663a-kube-api-access-4nff2\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.560822 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.560723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2acbbd40-8570-436c-9ec9-8e21403d663a-metrics-cert\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.560822 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.560773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2acbbd40-8570-436c-9ec9-8e21403d663a-manager-config\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.661832 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.661798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nff2\" (UniqueName: \"kubernetes.io/projected/2acbbd40-8570-436c-9ec9-8e21403d663a-kube-api-access-4nff2\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.661832 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.661834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2acbbd40-8570-436c-9ec9-8e21403d663a-metrics-cert\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.662037 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.661866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2acbbd40-8570-436c-9ec9-8e21403d663a-manager-config\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.662037 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.661923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acbbd40-8570-436c-9ec9-8e21403d663a-cert\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.662776 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.662681 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2acbbd40-8570-436c-9ec9-8e21403d663a-manager-config\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.664423 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.664403 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2acbbd40-8570-436c-9ec9-8e21403d663a-metrics-cert\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.664501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.664429 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2acbbd40-8570-436c-9ec9-8e21403d663a-cert\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.670565 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.670544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nff2\" (UniqueName: \"kubernetes.io/projected/2acbbd40-8570-436c-9ec9-8e21403d663a-kube-api-access-4nff2\") pod \"lws-controller-manager-7c889b46bf-x6kbw\" (UID: \"2acbbd40-8570-436c-9ec9-8e21403d663a\") " pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.803191 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.803157 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:39.933838 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.933798 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw"] Jul 09 23:32:39.937074 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:32:39.937043 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acbbd40_8570_436c_9ec9_8e21403d663a.slice/crio-90213b7d17628d2d0e21f50f49041fbbb663a4c2bc56f15c610d0df7d1068897 WatchSource:0}: Error finding container 90213b7d17628d2d0e21f50f49041fbbb663a4c2bc56f15c610d0df7d1068897: Status 404 returned error can't find the container with id 90213b7d17628d2d0e21f50f49041fbbb663a4c2bc56f15c610d0df7d1068897 Jul 09 23:32:39.958313 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:39.958287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" event={"ID":"2acbbd40-8570-436c-9ec9-8e21403d663a","Type":"ContainerStarted","Data":"90213b7d17628d2d0e21f50f49041fbbb663a4c2bc56f15c610d0df7d1068897"} Jul 09 23:32:42.972375 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:42.972294 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" event={"ID":"2acbbd40-8570-436c-9ec9-8e21403d663a","Type":"ContainerStarted","Data":"a99e9f141d5fad65231e17e2960559cdc2ce1f46be84b89052f8d789b110d435"} Jul 09 23:32:42.972715 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:42.972399 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:42.991633 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:42.991575 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" podStartSLOduration=1.321190769 podStartE2EDuration="3.991560786s" totalImagesPullingTime="2.670370017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:32:42.98971829 +0000 UTC m=+290.870470251" watchObservedRunningTime="2026-07-09 23:32:42.991560786 +0000 UTC m=+290.872312745" Jul 09 23:32:43.693072 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:43.693030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:43.785526 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:43.785492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:44.301031 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.300987 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:44.500790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.500757 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:44.609885 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.609805 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:44.651613 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.651585 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6"] Jul 09 23:32:44.654763 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.654746 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.657351 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.657331 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 23:32:44.657473 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.657366 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 23:32:44.657473 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.657458 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 23:32:44.657704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.657688 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8zrkh\"" Jul 09 23:32:44.657815 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.657709 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 23:32:44.672717 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.672690 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6"] Jul 09 23:32:44.703629 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.703581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7cfa21a5-a002-4495-9d8c-189edb2e0847-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.703761 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.703635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cfa21a5-a002-4495-9d8c-189edb2e0847-tmp\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.703761 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.703707 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndf7\" (UniqueName: \"kubernetes.io/projected/7cfa21a5-a002-4495-9d8c-189edb2e0847-kube-api-access-fndf7\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.703831 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.703760 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cfa21a5-a002-4495-9d8c-189edb2e0847-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.703831 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.703778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7cfa21a5-a002-4495-9d8c-189edb2e0847-manifests\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.804633 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.804595 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7cfa21a5-a002-4495-9d8c-189edb2e0847-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.804633 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.804635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cfa21a5-a002-4495-9d8c-189edb2e0847-tmp\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.804848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.804666 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndf7\" (UniqueName: \"kubernetes.io/projected/7cfa21a5-a002-4495-9d8c-189edb2e0847-kube-api-access-fndf7\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.804848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.804694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cfa21a5-a002-4495-9d8c-189edb2e0847-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.804848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.804712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7cfa21a5-a002-4495-9d8c-189edb2e0847-manifests\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.805102 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.805067 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7cfa21a5-a002-4495-9d8c-189edb2e0847-manifests\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.806849 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.806820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:44.807337 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.807313 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7cfa21a5-a002-4495-9d8c-189edb2e0847-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.807337 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.807313 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7cfa21a5-a002-4495-9d8c-189edb2e0847-tmp\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.807460 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.807365 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7cfa21a5-a002-4495-9d8c-189edb2e0847-webhook-cert\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.812474 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.812452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndf7\" (UniqueName: \"kubernetes.io/projected/7cfa21a5-a002-4495-9d8c-189edb2e0847-kube-api-access-fndf7\") pod \"opendatahub-operator-controller-manager-c55df6879-qbnb6\" (UID: \"7cfa21a5-a002-4495-9d8c-189edb2e0847\") " pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:44.907174 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.907095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:44.965079 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:44.965048 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:45.008230 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:45.008149 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:45.079626 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:45.079592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:45.107053 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:45.107016 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6"] Jul 09 23:32:45.116030 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:32:45.115988 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfa21a5_a002_4495_9d8c_189edb2e0847.slice/crio-3f5c7393f48b8fad8c62dc3e66f3befa0ecb3313b7c4a54f97af6f70e257ba93 WatchSource:0}: Error finding container 3f5c7393f48b8fad8c62dc3e66f3befa0ecb3313b7c4a54f97af6f70e257ba93: Status 404 returned error can't find the container with id 3f5c7393f48b8fad8c62dc3e66f3befa0ecb3313b7c4a54f97af6f70e257ba93 Jul 09 23:32:45.143915 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:45.143881 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:45.244397 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:45.244313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:45.986681 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:45.986647 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" event={"ID":"7cfa21a5-a002-4495-9d8c-189edb2e0847","Type":"ContainerStarted","Data":"3f5c7393f48b8fad8c62dc3e66f3befa0ecb3313b7c4a54f97af6f70e257ba93"} Jul 09 23:32:49.000867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:49.000831 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" event={"ID":"7cfa21a5-a002-4495-9d8c-189edb2e0847","Type":"ContainerStarted","Data":"22f7314a25b0979f1b953efa789f9ecdf0eed90e76b59d5d7e145819ae8b729a"} Jul 09 23:32:50.005984 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:50.005883 2562 generic.go:356] "Generic (PLEG): container finished" podID="7cfa21a5-a002-4495-9d8c-189edb2e0847" containerID="22f7314a25b0979f1b953efa789f9ecdf0eed90e76b59d5d7e145819ae8b729a" exitCode=0 Jul 09 23:32:50.005984 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:50.005968 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" event={"ID":"7cfa21a5-a002-4495-9d8c-189edb2e0847","Type":"ContainerDied","Data":"22f7314a25b0979f1b953efa789f9ecdf0eed90e76b59d5d7e145819ae8b729a"} Jul 09 23:32:51.012234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:51.012181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" event={"ID":"7cfa21a5-a002-4495-9d8c-189edb2e0847","Type":"ContainerStarted","Data":"301f575eb01fcb8c08fb1e7c882cf64e0f651d53542da5907ce5feb9a31e4b70"} Jul 09 23:32:51.012647 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:51.012294 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:32:51.034003 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:51.033954 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" podStartSLOduration=2.257476409 podStartE2EDuration="7.033937669s" totalImagesPullingTime="3.719733642s" totalInitContainerRuntime="1.056727618s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:32:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:32:51.032502138 +0000 UTC m=+298.913254108" watchObservedRunningTime="2026-07-09 23:32:51.033937669 +0000 UTC m=+298.914689628" Jul 09 23:32:51.262392 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:51.262303 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:52.450535 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:52.450489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:52.554238 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:52.554188 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:32:52.555482 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:52.555457 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:32:52.559417 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:52.559395 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 23:32:53.979699 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:53.979672 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7c889b46bf-x6kbw" Jul 09 23:32:55.903716 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:55.903678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:32:56.001939 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:32:56.001905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:01.678038 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:01.677999 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:01.778255 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:01.778223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:02.018696 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:02.018667 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-c55df6879-qbnb6" Jul 09 23:33:04.069186 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.069150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:04.171766 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.171734 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:04.462462 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.462376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:04.586393 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.586361 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6dff7df579-96c2g"] Jul 09 23:33:04.596422 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.596393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.600185 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.599984 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 23:33:04.600185 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.600163 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 09 23:33:04.600185 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.600167 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 23:33:04.600449 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.600180 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 09 23:33:04.601531 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.601502 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dff7df579-96c2g"] Jul 09 23:33:04.602057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.602035 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pbv6s\"" Jul 09 23:33:04.685519 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.685483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gn9p\" (UniqueName: \"kubernetes.io/projected/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-kube-api-access-4gn9p\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.685685 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.685526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-tmp\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.685685 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.685558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-tls-certs\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.741028 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.740955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:04.786956 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.786915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-tls-certs\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.787143 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.787009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gn9p\" (UniqueName: \"kubernetes.io/projected/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-kube-api-access-4gn9p\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.787143 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.787032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-tmp\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.789543 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.789517 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-tmp\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.789761 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.789738 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-tls-certs\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.797050 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.797023 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gn9p\" (UniqueName: \"kubernetes.io/projected/fc7a0613-0f47-48e0-8ccf-608fc7c0f535-kube-api-access-4gn9p\") pod \"kube-auth-proxy-6dff7df579-96c2g\" (UID: \"fc7a0613-0f47-48e0-8ccf-608fc7c0f535\") " pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.908739 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.908701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" Jul 09 23:33:04.969627 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:04.969584 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:05.061130 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.061100 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6dff7df579-96c2g"] Jul 09 23:33:05.063796 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:33:05.063770 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc7a0613_0f47_48e0_8ccf_608fc7c0f535.slice/crio-9e388e45339afa2580d00a70f8ea7abed961dbfd0eff9ae6e4515f36daf7b262 WatchSource:0}: Error finding container 9e388e45339afa2580d00a70f8ea7abed961dbfd0eff9ae6e4515f36daf7b262: Status 404 returned error can't find the container with id 9e388e45339afa2580d00a70f8ea7abed961dbfd0eff9ae6e4515f36daf7b262 Jul 09 23:33:05.065744 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.065727 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 23:33:05.069573 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.069546 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:05.071627 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.071604 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" event={"ID":"fc7a0613-0f47-48e0-8ccf-608fc7c0f535","Type":"ContainerStarted","Data":"9e388e45339afa2580d00a70f8ea7abed961dbfd0eff9ae6e4515f36daf7b262"} Jul 09 23:33:05.124964 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.124927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:05.330963 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.330933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:05.790578 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.790547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:05.886823 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.886788 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:05.985847 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:05.985805 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:06.185886 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:06.185795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:06.775067 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:06.775036 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:08.087638 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:08.087598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" event={"ID":"fc7a0613-0f47-48e0-8ccf-608fc7c0f535","Type":"ContainerStarted","Data":"c9401738e64b297ce0cb2f4af1dda3df6dbe1208542c7fb7e5353f88df67b94d"} Jul 09 23:33:08.103948 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:08.103897 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6dff7df579-96c2g" podStartSLOduration=1.285145225 podStartE2EDuration="4.103881429s" totalImagesPullingTime="2.818736204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:33:08.101880025 +0000 UTC m=+315.982631986" watchObservedRunningTime="2026-07-09 23:33:08.103881429 +0000 UTC m=+315.984633389" Jul 09 23:33:11.383167 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:11.383137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:11.486309 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:11.486272 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:12.111507 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:12.111471 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:12.291435 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:12.291397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:12.599965 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:12.599919 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:14.557952 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:14.557898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:14.643740 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:14.643704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:14.837539 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:14.837441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:14.995090 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:14.995027 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:15.095368 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:15.095271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:15.427650 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:15.427539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:15.821355 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:15.821323 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:15.921456 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:15.921414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:16.885351 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:16.885317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:16.948656 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:16.948625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:17.036955 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:17.036924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:17.089916 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:17.089883 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:33.861026 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:33.860983 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:33.960370 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:33.960322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:33:43.237607 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:33:43.237564 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:05.512479 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:05.512433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:05.613679 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:05.613624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:05.684460 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:05.684427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:06.442911 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:06.442868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:06.675182 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:06.675146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:28.700487 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:28.700455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:31.073187 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.073151 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75965c4cc9-lnxkv"] Jul 09 23:34:31.075400 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.075374 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.078046 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.078021 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 23:34:31.079094 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.079077 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 23:34:31.079234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.079116 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 23:34:31.079234 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.079129 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 23:34:31.079406 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.079311 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 23:34:31.079406 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.079315 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m97q4\"" Jul 09 23:34:31.079777 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.079426 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 23:34:31.079777 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.079453 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 23:34:31.083404 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.083388 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 23:34:31.088739 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.088718 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75965c4cc9-lnxkv"] Jul 09 23:34:31.134352 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.134321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-trusted-ca-bundle\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.134352 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.134361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l898p\" (UniqueName: \"kubernetes.io/projected/fa222157-09d7-4aa8-9577-5e34f775c20e-kube-api-access-l898p\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.134582 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.134433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa222157-09d7-4aa8-9577-5e34f775c20e-console-oauth-config\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.134582 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.134449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-console-config\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.134582 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.134464 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-oauth-serving-cert\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.134582 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.134485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-service-ca\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.134582 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.134536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa222157-09d7-4aa8-9577-5e34f775c20e-console-serving-cert\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.153641 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.153613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:31.235657 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.235624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa222157-09d7-4aa8-9577-5e34f775c20e-console-oauth-config\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.235657 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.235660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-console-config\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.235912 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.235677 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-oauth-serving-cert\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.235912 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.235871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-service-ca\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.235912 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.235908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa222157-09d7-4aa8-9577-5e34f775c20e-console-serving-cert\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.236072 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.235946 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-trusted-ca-bundle\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.236072 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.235978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l898p\" (UniqueName: \"kubernetes.io/projected/fa222157-09d7-4aa8-9577-5e34f775c20e-kube-api-access-l898p\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.236467 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.236442 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-console-config\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.236583 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.236484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-oauth-serving-cert\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.236704 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.236686 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-service-ca\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.236950 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.236934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa222157-09d7-4aa8-9577-5e34f775c20e-trusted-ca-bundle\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.238424 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.238401 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fa222157-09d7-4aa8-9577-5e34f775c20e-console-oauth-config\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.238600 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.238582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fa222157-09d7-4aa8-9577-5e34f775c20e-console-serving-cert\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.252943 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.252913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:31.254919 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.254900 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l898p\" (UniqueName: \"kubernetes.io/projected/fa222157-09d7-4aa8-9577-5e34f775c20e-kube-api-access-l898p\") pod \"console-75965c4cc9-lnxkv\" (UID: \"fa222157-09d7-4aa8-9577-5e34f775c20e\") " pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.385944 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.385852 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:31.501773 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.501742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:31.530603 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:31.530582 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75965c4cc9-lnxkv"] Jul 09 23:34:31.533154 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:34:31.533122 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa222157_09d7_4aa8_9577_5e34f775c20e.slice/crio-100df62ff87dddf7da22a2281114574b20996c7885c483616b67849415dea1aa WatchSource:0}: Error finding container 100df62ff87dddf7da22a2281114574b20996c7885c483616b67849415dea1aa: Status 404 returned error can't find the container with id 100df62ff87dddf7da22a2281114574b20996c7885c483616b67849415dea1aa Jul 09 23:34:32.447661 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:32.447625 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75965c4cc9-lnxkv" event={"ID":"fa222157-09d7-4aa8-9577-5e34f775c20e","Type":"ContainerStarted","Data":"a9a6e5915dd0060b11e2e77a25a573afb56728f6d7413466061939c9d9cee69c"} Jul 09 23:34:32.448046 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:32.447673 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75965c4cc9-lnxkv" event={"ID":"fa222157-09d7-4aa8-9577-5e34f775c20e","Type":"ContainerStarted","Data":"100df62ff87dddf7da22a2281114574b20996c7885c483616b67849415dea1aa"} Jul 09 23:34:32.472220 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:32.471794 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75965c4cc9-lnxkv" podStartSLOduration=1.471770893 podStartE2EDuration="1.471770893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:32.465147616 +0000 UTC m=+400.345899579" watchObservedRunningTime="2026-07-09 23:34:32.471770893 +0000 UTC m=+400.352522856" Jul 09 23:34:36.702708 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:36.702647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:37.283846 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:37.283811 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:37.755070 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:37.754976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:40.902857 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:40.902818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:41.386826 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:41.386789 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:41.387002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:41.386849 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:41.391548 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:41.391527 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:41.488768 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:41.488740 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75965c4cc9-lnxkv" Jul 09 23:34:41.668646 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:41.668555 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:41.773722 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:41.773692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:41.946456 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:41.946364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:42.102920 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:42.102873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:42.990303 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:42.990271 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5"] Jul 09 23:34:42.992680 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:42.992663 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" Jul 09 23:34:42.995139 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:42.995120 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-g25nf\"" Jul 09 23:34:42.996183 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:42.996166 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 23:34:42.996300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:42.996213 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 23:34:42.996300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:42.996171 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 23:34:43.003238 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.003215 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5"] Jul 09 23:34:43.148380 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.148338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs84k\" (UniqueName: \"kubernetes.io/projected/b516a848-78cf-4d78-850e-cd42b69fbca4-kube-api-access-gs84k\") pod \"dns-operator-controller-manager-864dd78c4c-7tbp5\" (UID: \"b516a848-78cf-4d78-850e-cd42b69fbca4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" Jul 09 23:34:43.248993 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.248916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs84k\" (UniqueName: \"kubernetes.io/projected/b516a848-78cf-4d78-850e-cd42b69fbca4-kube-api-access-gs84k\") pod \"dns-operator-controller-manager-864dd78c4c-7tbp5\" (UID: \"b516a848-78cf-4d78-850e-cd42b69fbca4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" Jul 09 23:34:43.257083 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.257053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs84k\" (UniqueName: \"kubernetes.io/projected/b516a848-78cf-4d78-850e-cd42b69fbca4-kube-api-access-gs84k\") pod \"dns-operator-controller-manager-864dd78c4c-7tbp5\" (UID: \"b516a848-78cf-4d78-850e-cd42b69fbca4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" Jul 09 23:34:43.304063 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.304016 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" Jul 09 23:34:43.431432 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.431406 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5"] Jul 09 23:34:43.433875 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:34:43.433839 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb516a848_78cf_4d78_850e_cd42b69fbca4.slice/crio-4c93c553374b938b63dc2a77172ee1a1b3dd18069ac915bba819680c4ffe5207 WatchSource:0}: Error finding container 4c93c553374b938b63dc2a77172ee1a1b3dd18069ac915bba819680c4ffe5207: Status 404 returned error can't find the container with id 4c93c553374b938b63dc2a77172ee1a1b3dd18069ac915bba819680c4ffe5207 Jul 09 23:34:43.495017 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.494980 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" event={"ID":"b516a848-78cf-4d78-850e-cd42b69fbca4","Type":"ContainerStarted","Data":"4c93c553374b938b63dc2a77172ee1a1b3dd18069ac915bba819680c4ffe5207"} Jul 09 23:34:43.765096 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:43.765063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:45.037438 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.037402 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h"] Jul 09 23:34:45.039696 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.039675 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.042247 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.042157 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 09 23:34:45.042247 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.042191 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xk9b9\"" Jul 09 23:34:45.042411 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.042287 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 09 23:34:45.043534 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.043501 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:45.047701 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.047661 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h"] Jul 09 23:34:45.166996 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.166960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3303cbae-7fa3-4d62-bf8a-6c8720231294-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.166996 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.166999 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw2sr\" (UniqueName: \"kubernetes.io/projected/3303cbae-7fa3-4d62-bf8a-6c8720231294-kube-api-access-dw2sr\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.167276 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.167041 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303cbae-7fa3-4d62-bf8a-6c8720231294-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.267951 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.267913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303cbae-7fa3-4d62-bf8a-6c8720231294-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.268161 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.268052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3303cbae-7fa3-4d62-bf8a-6c8720231294-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.268161 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.268081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw2sr\" (UniqueName: \"kubernetes.io/projected/3303cbae-7fa3-4d62-bf8a-6c8720231294-kube-api-access-dw2sr\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.268161 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:34:45.268127 2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 09 23:34:45.268358 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:34:45.268225 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3303cbae-7fa3-4d62-bf8a-6c8720231294-plugin-serving-cert podName:3303cbae-7fa3-4d62-bf8a-6c8720231294 nodeName:}" failed. No retries permitted until 2026-07-09 23:34:45.768186195 +0000 UTC m=+413.648938142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/3303cbae-7fa3-4d62-bf8a-6c8720231294-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-c4s2h" (UID: "3303cbae-7fa3-4d62-bf8a-6c8720231294") : secret "plugin-serving-cert" not found Jul 09 23:34:45.268842 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.268825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3303cbae-7fa3-4d62-bf8a-6c8720231294-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.278844 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.278817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw2sr\" (UniqueName: \"kubernetes.io/projected/3303cbae-7fa3-4d62-bf8a-6c8720231294-kube-api-access-dw2sr\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.328343 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.328255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:45.773609 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.773515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303cbae-7fa3-4d62-bf8a-6c8720231294-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.776162 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.776138 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/3303cbae-7fa3-4d62-bf8a-6c8720231294-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-c4s2h\" (UID: \"3303cbae-7fa3-4d62-bf8a-6c8720231294\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:45.961599 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:45.961556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" Jul 09 23:34:46.089925 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:46.089901 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h"] Jul 09 23:34:46.091978 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:34:46.091957 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3303cbae_7fa3_4d62_bf8a_6c8720231294.slice/crio-63b8641217bb024f2b611cf9c31cfb6be2205472d5593b1e1c31fb579b68498f WatchSource:0}: Error finding container 63b8641217bb024f2b611cf9c31cfb6be2205472d5593b1e1c31fb579b68498f: Status 404 returned error can't find the container with id 63b8641217bb024f2b611cf9c31cfb6be2205472d5593b1e1c31fb579b68498f Jul 09 23:34:46.513906 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:46.513845 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" event={"ID":"3303cbae-7fa3-4d62-bf8a-6c8720231294","Type":"ContainerStarted","Data":"63b8641217bb024f2b611cf9c31cfb6be2205472d5593b1e1c31fb579b68498f"} Jul 09 23:34:46.515350 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:46.515324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" event={"ID":"b516a848-78cf-4d78-850e-cd42b69fbca4","Type":"ContainerStarted","Data":"eaf38110b0001c1fdd5a28af80401bdb174492aa3d162da9889e1c87e3ca3db9"} Jul 09 23:34:46.515476 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:46.515450 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" Jul 09 23:34:46.537146 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:46.537099 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" podStartSLOduration=2.518585039 podStartE2EDuration="4.53708506s" totalImagesPullingTime="2.018500021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:34:46.536517376 +0000 UTC m=+414.417269328" watchObservedRunningTime="2026-07-09 23:34:46.53708506 +0000 UTC m=+414.417837019" Jul 09 23:34:46.545153 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:46.545117 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:47.576933 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:47.576884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:48.699426 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:48.699388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:48.756300 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:48.756196 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:57.525441 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:57.525407 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7tbp5" Jul 09 23:34:58.927555 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:58.927513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:34:59.027672 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:34:59.027638 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:01.365136 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:01.365101 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:01.465005 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:01.464938 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:05.707480 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:05.707443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:05.810338 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:05.810296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:08.110663 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:08.110623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:08.202192 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:08.202156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:09.633645 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:09.633617 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" event={"ID":"3303cbae-7fa3-4d62-bf8a-6c8720231294","Type":"ContainerStarted","Data":"bc01367beb53e1994e01c87848adfc73705dae123d45259c01a6af075342f014"} Jul 09 23:35:09.648062 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:09.648012 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-c4s2h" podStartSLOduration=1.6072416889999999 podStartE2EDuration="24.647998802s" totalImagesPullingTime="23.040757113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:34:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:09.646775196 +0000 UTC m=+437.527527158" watchObservedRunningTime="2026-07-09 23:35:09.647998802 +0000 UTC m=+437.528750762" Jul 09 23:35:16.912605 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:16.912516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:17.019616 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:17.019585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:19.373872 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:19.373832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:19.473856 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:19.473820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:23.667880 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:23.667844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:23.765224 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:23.765160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:23.868460 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:23.868427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:25.809120 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:25.809091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:25.923376 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:25.923323 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:25.982682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:25.982646 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:26.308991 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.308959 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:35:26.351601 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.351565 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:35:26.351601 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.351599 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:35:26.351803 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.351684 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.354087 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.354070 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 23:35:26.450074 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.450037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cedcc116-847f-4f50-b3ed-182fc7e813ff-config-file\") pod \"limitador-limitador-54d6d988d5-q7r2l\" (UID: \"cedcc116-847f-4f50-b3ed-182fc7e813ff\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.450074 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.450079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8895\" (UniqueName: \"kubernetes.io/projected/cedcc116-847f-4f50-b3ed-182fc7e813ff-kube-api-access-l8895\") pod \"limitador-limitador-54d6d988d5-q7r2l\" (UID: \"cedcc116-847f-4f50-b3ed-182fc7e813ff\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.550883 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.550848 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cedcc116-847f-4f50-b3ed-182fc7e813ff-config-file\") pod \"limitador-limitador-54d6d988d5-q7r2l\" (UID: \"cedcc116-847f-4f50-b3ed-182fc7e813ff\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.550883 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.550888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8895\" (UniqueName: \"kubernetes.io/projected/cedcc116-847f-4f50-b3ed-182fc7e813ff-kube-api-access-l8895\") pod \"limitador-limitador-54d6d988d5-q7r2l\" (UID: \"cedcc116-847f-4f50-b3ed-182fc7e813ff\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.551508 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.551491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cedcc116-847f-4f50-b3ed-182fc7e813ff-config-file\") pod \"limitador-limitador-54d6d988d5-q7r2l\" (UID: \"cedcc116-847f-4f50-b3ed-182fc7e813ff\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.558878 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.558848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8895\" (UniqueName: \"kubernetes.io/projected/cedcc116-847f-4f50-b3ed-182fc7e813ff-kube-api-access-l8895\") pod \"limitador-limitador-54d6d988d5-q7r2l\" (UID: \"cedcc116-847f-4f50-b3ed-182fc7e813ff\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.637946 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.637830 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:26.662056 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.662020 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:26.692537 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.692489 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:26.793452 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:26.793426 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:35:26.796130 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:35:26.796100 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcedcc116_847f_4f50_b3ed_182fc7e813ff.slice/crio-81775031d8cad23b31fc03ee90e311db0b66179c80723d0efa1e4a175b2eb13b WatchSource:0}: Error finding container 81775031d8cad23b31fc03ee90e311db0b66179c80723d0efa1e4a175b2eb13b: Status 404 returned error can't find the container with id 81775031d8cad23b31fc03ee90e311db0b66179c80723d0efa1e4a175b2eb13b Jul 09 23:35:27.322761 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:27.322720 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:27.423021 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:27.422968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:27.714032 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:27.713937 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" event={"ID":"cedcc116-847f-4f50-b3ed-182fc7e813ff","Type":"ContainerStarted","Data":"81775031d8cad23b31fc03ee90e311db0b66179c80723d0efa1e4a175b2eb13b"} Jul 09 23:35:29.725421 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:29.725383 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" event={"ID":"cedcc116-847f-4f50-b3ed-182fc7e813ff","Type":"ContainerStarted","Data":"f3c0b52ec2a2607293cef335e45bd84fb9430a40f1e91d50d9619f45d71dfe66"} Jul 09 23:35:29.725804 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:29.725563 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:29.742330 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:29.742284 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" podStartSLOduration=1.046290911 podStartE2EDuration="3.742267961s" totalImagesPullingTime="2.69597705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:35:29.740297393 +0000 UTC m=+457.621049354" watchObservedRunningTime="2026-07-09 23:35:29.742267961 +0000 UTC m=+457.623019920" Jul 09 23:35:37.334855 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:37.334816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:37.444500 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:37.444453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:39.767358 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:39.767309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:39.822002 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:39.821952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:40.731069 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:40.731043 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q7r2l" Jul 09 23:35:41.911574 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:41.911543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:42.010625 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:42.010578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:42.361065 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:42.361015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:50.648125 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:50.648093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:54.927878 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:54.927842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:35:55.027959 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:35:55.027919 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:00.790243 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:00.790187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:00.890390 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:00.890357 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:00.977091 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:00.977063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:01.106163 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:01.106046 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:04.349715 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:04.349674 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:04.486033 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:04.485998 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:33.853489 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:33.853451 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:33.953219 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:33.953171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:38.313367 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:38.313332 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:38.736807 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:38.736728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:39.058871 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:39.058835 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:39.258783 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:39.258738 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:39.361526 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:39.361432 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:39.452345 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:39.452306 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:36:48.815805 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:48.815678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:48.915322 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:48.915287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:52.992660 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:52.992624 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9"] Jul 09 23:36:52.995885 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:52.995867 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:52.999442 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:52.999412 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 09 23:36:52.999551 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:52.999476 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-28khl\"" Jul 09 23:36:52.999551 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:52.999504 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 09 23:36:52.999551 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:52.999546 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 09 23:36:53.007317 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.007296 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9"] Jul 09 23:36:53.018464 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.018438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:53.113848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.113815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.113848 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.113847 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405f9927-c151-495d-8fe1-afc009a8e57f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.114088 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.113869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.114088 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.113933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.114088 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.114042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.114088 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.114062 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc48r\" (UniqueName: \"kubernetes.io/projected/405f9927-c151-495d-8fe1-afc009a8e57f-kube-api-access-tc48r\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215197 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc48r\" (UniqueName: \"kubernetes.io/projected/405f9927-c151-495d-8fe1-afc009a8e57f-kube-api-access-tc48r\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215391 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215391 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215243 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405f9927-c151-495d-8fe1-afc009a8e57f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215391 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215391 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215391 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215663 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215663 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.215759 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.215725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.217646 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.217625 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/405f9927-c151-495d-8fe1-afc009a8e57f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.217940 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.217923 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/405f9927-c151-495d-8fe1-afc009a8e57f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.223182 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.223160 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc48r\" (UniqueName: \"kubernetes.io/projected/405f9927-c151-495d-8fe1-afc009a8e57f-kube-api-access-tc48r\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9\" (UID: \"405f9927-c151-495d-8fe1-afc009a8e57f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.306270 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.306235 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:36:53.436531 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.436507 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9"] Jul 09 23:36:53.439115 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:36:53.439085 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405f9927_c151_495d_8fe1_afc009a8e57f.slice/crio-608efacbcd29fffe2fd8e8b74135dd905af85377cd955607719b4dfac2bbd71f WatchSource:0}: Error finding container 608efacbcd29fffe2fd8e8b74135dd905af85377cd955607719b4dfac2bbd71f: Status 404 returned error can't find the container with id 608efacbcd29fffe2fd8e8b74135dd905af85377cd955607719b4dfac2bbd71f Jul 09 23:36:53.596674 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:53.596594 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:36:54.090746 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:54.090706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" event={"ID":"405f9927-c151-495d-8fe1-afc009a8e57f","Type":"ContainerStarted","Data":"608efacbcd29fffe2fd8e8b74135dd905af85377cd955607719b4dfac2bbd71f"} Jul 09 23:36:55.096058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:55.096019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" event={"ID":"405f9927-c151-495d-8fe1-afc009a8e57f","Type":"ContainerStarted","Data":"bb6a7c64268b0c6aec8bfd7abd808081eadc45066dd7cf28efc8b1aff79a7c17"} Jul 09 23:36:55.113297 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:55.113237 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" podStartSLOduration=1.950459522 podStartE2EDuration="3.1132208s" totalImagesPullingTime="1.162761278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:36:55.111823691 +0000 UTC m=+542.992575651" watchObservedRunningTime="2026-07-09 23:36:55.1132208 +0000 UTC m=+542.993972761" Jul 09 23:36:57.100941 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:57.100912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:36:57.989097 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:36:57.989063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:37:03.307421 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:03.307388 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:37:03.307828 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:03.307456 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:37:03.319669 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:03.319639 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:37:04.147496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:04.147465 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9" Jul 09 23:37:16.279115 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.279083 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk"] Jul 09 23:37:16.282387 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.282369 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.284691 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.284673 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 09 23:37:16.291193 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.291167 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk"] Jul 09 23:37:16.318849 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.318823 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.319025 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.318882 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82c6f2-11b0-4b41-a18a-7d50d7314000-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.319025 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.318941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.319025 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.319011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgb2\" (UniqueName: \"kubernetes.io/projected/ba82c6f2-11b0-4b41-a18a-7d50d7314000-kube-api-access-glgb2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.319190 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.319075 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.319190 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.319102 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.378557 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.378523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:37:16.419774 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.419738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420001 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.419795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgb2\" (UniqueName: \"kubernetes.io/projected/ba82c6f2-11b0-4b41-a18a-7d50d7314000-kube-api-access-glgb2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420001 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.419834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420001 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.419860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420001 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.419878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420001 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.419912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82c6f2-11b0-4b41-a18a-7d50d7314000-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420284 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.420270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420348 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.420308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.420427 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.420405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.422265 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.422248 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba82c6f2-11b0-4b41-a18a-7d50d7314000-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.422492 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.422476 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba82c6f2-11b0-4b41-a18a-7d50d7314000-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.427526 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.427506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgb2\" (UniqueName: \"kubernetes.io/projected/ba82c6f2-11b0-4b41-a18a-7d50d7314000-kube-api-access-glgb2\") pod \"e2e-trlp-test-simulated-kserve-598466548d-nqhjk\" (UID: \"ba82c6f2-11b0-4b41-a18a-7d50d7314000\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.592347 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.592236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:16.723121 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:16.723091 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk"] Jul 09 23:37:16.726479 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:37:16.726449 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba82c6f2_11b0_4b41_a18a_7d50d7314000.slice/crio-28347a2848583d3ae597e645b05f10efeab26448e69008251b95271a49e929ca WatchSource:0}: Error finding container 28347a2848583d3ae597e645b05f10efeab26448e69008251b95271a49e929ca: Status 404 returned error can't find the container with id 28347a2848583d3ae597e645b05f10efeab26448e69008251b95271a49e929ca Jul 09 23:37:17.194842 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:17.194752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" event={"ID":"ba82c6f2-11b0-4b41-a18a-7d50d7314000","Type":"ContainerStarted","Data":"015ffae5a3e20dc62e82ed6a539a368cec799e0d63036b019a42e94bb91935af"} Jul 09 23:37:17.194842 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:17.194816 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" event={"ID":"ba82c6f2-11b0-4b41-a18a-7d50d7314000","Type":"ContainerStarted","Data":"28347a2848583d3ae597e645b05f10efeab26448e69008251b95271a49e929ca"} Jul 09 23:37:17.213520 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:17.213470 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" podStartSLOduration=1.030373815 podStartE2EDuration="1.213454763s" totalImagesPullingTime="183.080948ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:37:17.211630805 +0000 UTC m=+565.092382783" watchObservedRunningTime="2026-07-09 23:37:17.213454763 +0000 UTC m=+565.094206723" Jul 09 23:37:17.678365 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:17.678332 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:37:26.592547 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:26.592506 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:26.592983 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:26.592562 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:26.605252 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:26.605226 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:27.250980 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:27.250947 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-nqhjk" Jul 09 23:37:37.725687 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:37.725647 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:37:39.284172 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:39.284138 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:37:50.080934 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.080883 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w"] Jul 09 23:37:50.084286 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.084264 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.086859 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.086840 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 09 23:37:50.095196 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.095175 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w"] Jul 09 23:37:50.114777 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.114753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:37:50.227578 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.227545 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.227737 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.227593 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.227737 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.227635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.227737 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.227655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.227737 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.227719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs8hz\" (UniqueName: \"kubernetes.io/projected/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-kube-api-access-fs8hz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.227875 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.227787 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.328834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.328795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.328834 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.328839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.329047 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.328860 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.329047 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.328880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.329047 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.328936 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs8hz\" (UniqueName: \"kubernetes.io/projected/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-kube-api-access-fs8hz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.329047 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.328978 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.329292 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.329269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.329346 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.329311 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.329386 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.329356 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.331392 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.331347 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.331635 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.331614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.336577 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.336552 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs8hz\" (UniqueName: \"kubernetes.io/projected/5a9c2964-a3c7-4855-a56f-c0d2e27845bc-kube-api-access-fs8hz\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vpp9w\" (UID: \"5a9c2964-a3c7-4855-a56f-c0d2e27845bc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.394801 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.394764 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:37:50.526478 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:50.526444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w"] Jul 09 23:37:50.529609 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:37:50.529583 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9c2964_a3c7_4855_a56f_c0d2e27845bc.slice/crio-d5cf14ec58d21248dac5fd89f2668ecd0c5ffb367117cf716b8efc0c0b9e8453 WatchSource:0}: Error finding container d5cf14ec58d21248dac5fd89f2668ecd0c5ffb367117cf716b8efc0c0b9e8453: Status 404 returned error can't find the container with id d5cf14ec58d21248dac5fd89f2668ecd0c5ffb367117cf716b8efc0c0b9e8453 Jul 09 23:37:51.349217 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:51.349174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" event={"ID":"5a9c2964-a3c7-4855-a56f-c0d2e27845bc","Type":"ContainerStarted","Data":"239074b8bcdf4c6792d83cebbcdefd801c6c9a4edfadfc6bf47e3ec5a10e41f3"} Jul 09 23:37:51.349616 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:51.349253 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" event={"ID":"5a9c2964-a3c7-4855-a56f-c0d2e27845bc","Type":"ContainerStarted","Data":"d5cf14ec58d21248dac5fd89f2668ecd0c5ffb367117cf716b8efc0c0b9e8453"} Jul 09 23:37:51.365812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:51.365761 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" podStartSLOduration=1.189859633 podStartE2EDuration="1.365745827s" totalImagesPullingTime="175.886194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:37:51.364284928 +0000 UTC m=+599.245036887" watchObservedRunningTime="2026-07-09 23:37:51.365745827 +0000 UTC m=+599.246497764" Jul 09 23:37:52.386402 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:52.386368 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:37:52.581614 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:52.581586 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:37:52.582058 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:52.582040 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:37:59.276981 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:37:59.276944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:38:00.395085 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:00.395054 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:38:00.395492 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:00.395110 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:38:00.407538 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:00.407516 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:38:01.404642 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:01.404612 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vpp9w" Jul 09 23:38:03.282504 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:03.282471 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:38:21.351189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:21.351105 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:38:21.398772 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:21.398745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:38:48.795147 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:48.795085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:38:48.859952 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:48.859908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:38:49.918582 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:49.918542 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:38:50.019066 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:38:50.019032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:39:02.652100 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:02.652067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:39:02.751660 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:02.751629 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:39:03.510740 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:03.510702 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:39:03.559875 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:03.559831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:39:18.279138 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:18.279102 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:39:28.081525 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:28.081489 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:39:39.185928 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:39.185892 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:39:47.375751 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:47.375656 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:39:58.279464 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:39:58.279429 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:40:00.356478 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:00.356446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:01.571533 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:01.571502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:19.398020 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:19.397985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:25.982917 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:25.982882 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:40:39.174985 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:39.174949 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:40:44.181068 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:44.181030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:44.279136 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:44.279102 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:44.396745 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:44.396714 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:50.006798 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:50.006768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:50.108109 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:50.108078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:50.207150 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:50.207103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:50.331404 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:50.331322 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:55.921121 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:55.921085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:55.976068 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:55.976037 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:58.636319 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:58.636263 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:40:58.690620 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:40:58.690584 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:41:14.281382 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:41:14.281297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:41:23.781018 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:41:23.780985 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:41:23.941168 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:41:23.941133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:41:24.432014 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:41:24.431983 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:41:33.584623 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:41:33.584586 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:41:41.083853 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:41:41.083816 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:41:52.776347 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:41:52.776316 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:42:04.987719 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:42:04.987688 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:42:16.279279 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:42:16.279234 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:42:26.583091 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:42:26.583058 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:42:50.390173 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:42:50.390093 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:42:52.615700 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:42:52.615663 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:42:52.617670 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:42:52.617648 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:43:02.781182 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:43:02.781152 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:43:04.549489 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:43:04.549453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:43:26.074279 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:43:26.074244 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:43:31.661000 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:43:31.660956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:43:31.763867 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:43:31.763829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:43:46.580978 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:43:46.580940 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:43:57.276345 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:43:57.276312 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:44:07.286605 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:44:07.286568 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:44:17.082984 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:44:17.082902 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:44:27.377742 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:44:27.377701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:44:36.983146 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:44:36.983113 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:46:24.185123 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:46:24.185091 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:46:35.383840 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:46:35.383798 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:46:45.976249 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:46:45.976193 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:46:58.288273 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:46:58.288237 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:47:08.179948 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:08.179908 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:47:15.514641 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:15.514609 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:47:19.984709 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:19.984671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:47:28.281656 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:28.281623 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:47:41.292352 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:41.292318 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:47:51.074579 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:51.074544 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:47:52.643750 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:52.643712 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:47:52.646316 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:52.646298 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:47:59.877347 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:47:59.877309 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:48:13.781150 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:48:13.781119 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:48:31.646512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:48:31.646467 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:48:41.185746 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:48:41.185708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:48:51.277689 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:48:51.277593 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:49:01.081750 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:49:01.081713 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:49:18.482233 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:49:18.482181 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:49:28.141318 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:49:28.141286 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:49:37.875927 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:49:37.875891 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:49:46.086146 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:49:46.086108 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:49:55.783047 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:49:55.783001 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:50:04.582006 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:50:04.581969 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:50:18.284432 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:50:18.284356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:50:28.479684 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:50:28.479645 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:50:38.377994 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:50:38.377958 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:50:51.181134 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:50:51.181098 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:51:00.385884 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:00.385834 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:51:07.679792 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:07.679757 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:51:09.970987 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:09.970955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:51:11.376606 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:11.376574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:51:23.076389 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:23.076358 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:51:27.662109 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:27.662072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:51:34.880353 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:34.880318 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:51:46.086895 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:46.086812 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:51:49.056255 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:49.056220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:51:55.483847 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:51:55.483807 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:52:00.696691 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:00.696658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:06.196129 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:06.196092 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:52:08.939942 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:08.939909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:14.003652 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:14.003620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:16.808141 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:16.808097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:16.904853 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:16.904797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:17.255004 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:17.254916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:18.585546 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:18.585501 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:52:22.933151 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:22.933122 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:22.991785 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:22.991753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:23.129194 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:23.129150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:23.186962 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:23.186873 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:24.324189 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:24.324145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:24.441394 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:24.441365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:24.539385 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:24.539350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:24.638524 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:24.638431 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:24.943534 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:24.943459 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:25.037093 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:25.037063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:26.668961 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:26.668929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:29.309232 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:29.309183 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:29.493266 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:29.493229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:29.590987 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:29.590896 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:29.661014 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:29.660985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:32.719043 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:32.719001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:35.267795 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:35.267758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:35.321073 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:35.321038 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:35.460580 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:35.460553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:35.513411 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:35.513377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:36.618826 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:36.618797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:36.700287 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:36.700255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:36.834428 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:36.834391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:37.355423 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:37.355388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:37.456332 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:37.456301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:38.386663 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:38.386628 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:52:39.207040 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:39.207009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:41.698778 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:41.698740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:42.033325 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:42.033289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:48.581982 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:48.581949 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:48.631074 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:48.631011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:49.825525 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:49.825481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:49.981588 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:49.981552 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:52:52.670580 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:52.670555 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:52:52.673671 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:52.673653 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:52:53.456004 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:53.455967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:53.558809 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:53.558776 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:53.679539 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:53.679510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:53.838671 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:53.834311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:54.760903 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:54.760866 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:54.855988 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:54.855957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:54.914714 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:54.914682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:55.113672 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:55.113582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:55.404778 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:55.404691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:55.504807 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:55.504773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:56.952649 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:56.952623 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:59.217934 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:59.217903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:59.381983 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:59.381943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:59.481106 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:59.481020 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:52:59.598862 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:52:59.598830 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:01.023550 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:01.023518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:05.258948 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:05.258909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:05.312809 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:05.312776 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:05.709938 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:05.709846 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:05.977886 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:05.977810 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:06.075193 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:06.075152 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:07.237258 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:07.237226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:12.761321 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:12.761270 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:12.822057 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:12.822010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:12.940996 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:12.940960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:13.061883 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:13.061799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:13.468302 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:13.468191 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:19.051251 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:19.051194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:39.416449 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:39.416412 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:45.023995 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:45.023948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:45.084711 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:45.084676 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:45.162799 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:45.162748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:53:45.262063 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:53:45.262026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:01.104524 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:01.104487 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:06.910964 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:06.910927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:06.971083 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:06.971048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:07.049995 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:07.049966 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:07.150048 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:07.150014 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:17.593244 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:17.593187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:18.381268 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:18.381232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:23.914446 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:23.914407 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:23.975849 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:23.975814 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:24.049592 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:24.049557 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:24.149708 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:24.149677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:24.251184 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:24.251105 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:24.947512 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:24.947476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:25.089738 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:25.089705 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:25.185520 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:25.185491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:26.670258 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:26.670227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:27.578344 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:27.578306 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:54:32.217854 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:32.217819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:32.318386 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:32.318349 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:32.873452 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:32.873421 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:35.384635 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:35.384601 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:54:38.539294 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:38.539260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:38.641055 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:38.641018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:38.758941 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:38.758913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:39.086594 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:39.086560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:39.294229 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:39.294172 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:42.410115 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:42.410078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:42.488675 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:42.488648 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:42.551075 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:42.551033 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:42.933389 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:42.933355 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:43.034122 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:43.034078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:46.874383 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:46.874305 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:47.519552 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:47.519515 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:47.613415 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:47.613385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:47.922776 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:47.922690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:49.483958 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:49.483879 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:54:53.692895 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:53.692860 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:53.747999 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:53.747962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:54.148939 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:54.148905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:54.547091 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:54.547056 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:54:54.644831 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:54:54.644795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:00.207001 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:00.206950 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:00.261991 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:00.261963 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:00.447260 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:00.447186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:00.674065 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:00.674039 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:00.797255 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:00.797220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:00.898154 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:00.898117 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:00.999724 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:00.999637 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:55:01.003186 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:01.003164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:06.659547 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:06.659513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:06.808717 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:06.808678 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:06.903044 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:06.903010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:07.006096 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:07.006019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:12.575363 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:12.575324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:12.630912 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:12.630876 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:19.301794 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:19.301758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:19.630077 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:19.629992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:19.730246 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:19.730210 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:22.662781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:22.662752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:22.740517 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:22.740483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:22.805085 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:22.805050 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:23.087606 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:23.087575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:23.302812 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:23.302781 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:23.404125 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:23.404033 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:24.245251 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:24.245219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:24.345280 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:24.345248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:25.042627 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:25.042588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:25.144638 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:25.144606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:27.122454 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:27.122419 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:30.385559 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:30.385521 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:55:44.685526 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.685488 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp"] Jul 09 23:55:44.688871 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.688854 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.692417 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.692397 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7b0dea16\"/\"default-dockercfg-k7k8x\"" Jul 09 23:55:44.692417 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.692403 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7b0dea16\"/\"kube-root-ca.crt\"" Jul 09 23:55:44.692597 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.692397 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7b0dea16\"/\"e2e-lifecycle-model-7b0dea16-kserve-self-signed-certs\"" Jul 09 23:55:44.692597 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.692445 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7b0dea16\"/\"openshift-service-ca.crt\"" Jul 09 23:55:44.696260 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.696230 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp"] Jul 09 23:55:44.761339 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.761304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-model-cache\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.761339 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.761334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-tmp-dir\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.761533 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.761359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbbm8\" (UniqueName: \"kubernetes.io/projected/86482ac6-f516-47ca-9431-39ae80ab7ed3-kube-api-access-sbbm8\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.761533 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.761447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86482ac6-f516-47ca-9431-39ae80ab7ed3-tls-certs\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.761533 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.761499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-home\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.761628 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.761574 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-dshm\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.773276 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.773252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:55:44.862361 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-dshm\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862541 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862372 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-model-cache\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862541 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-tmp-dir\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862541 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbbm8\" (UniqueName: \"kubernetes.io/projected/86482ac6-f516-47ca-9431-39ae80ab7ed3-kube-api-access-sbbm8\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862541 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86482ac6-f516-47ca-9431-39ae80ab7ed3-tls-certs\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862541 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862508 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-home\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862810 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-model-cache\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862890 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862869 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-tmp-dir\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.862932 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.862887 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-home\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.864881 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.864855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-dshm\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.865078 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.865063 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86482ac6-f516-47ca-9431-39ae80ab7ed3-tls-certs\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.869569 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.869545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbbm8\" (UniqueName: \"kubernetes.io/projected/86482ac6-f516-47ca-9431-39ae80ab7ed3-kube-api-access-sbbm8\") pod \"e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:44.998829 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:44.998743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:45.131427 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:45.131402 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp"] Jul 09 23:55:45.133421 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:55:45.133393 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86482ac6_f516_47ca_9431_39ae80ab7ed3.slice/crio-3494fb02e6c31a585537389dfb20c582102e5df73845f02c625f04300c50990b WatchSource:0}: Error finding container 3494fb02e6c31a585537389dfb20c582102e5df73845f02c625f04300c50990b: Status 404 returned error can't find the container with id 3494fb02e6c31a585537389dfb20c582102e5df73845f02c625f04300c50990b Jul 09 23:55:45.297790 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:45.297751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" event={"ID":"86482ac6-f516-47ca-9431-39ae80ab7ed3","Type":"ContainerStarted","Data":"0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c"} Jul 09 23:55:45.297966 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:45.297803 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" event={"ID":"86482ac6-f516-47ca-9431-39ae80ab7ed3","Type":"ContainerStarted","Data":"3494fb02e6c31a585537389dfb20c582102e5df73845f02c625f04300c50990b"} Jul 09 23:55:45.313368 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:45.313321 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" podStartSLOduration=1.313307703 podStartE2EDuration="1.313307703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:55:45.312502595 +0000 UTC m=+1673.193254559" watchObservedRunningTime="2026-07-09 23:55:45.313307703 +0000 UTC m=+1673.194059663" Jul 09 23:55:47.880515 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:47.880474 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:55:54.999280 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:54.999244 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:54.999674 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:54.999324 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:55.013678 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:55.013648 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:55:55.356464 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:55:55.356430 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:56:07.182022 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:07.181989 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:56:09.651564 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:09.651530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:56:10.042091 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:10.042063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:56:15.700668 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:15.700583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:56:20.685503 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:20.685465 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp"] Jul 09 23:56:20.730096 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:20.730062 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:56:20.784114 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:20.784076 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:56:35.707781 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:35.707723 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" podUID="86482ac6-f516-47ca-9431-39ae80ab7ed3" containerName="main" containerID="cri-o://0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c" gracePeriod=45 Jul 09 23:56:35.963427 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:35.963359 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:56:36.158807 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.158766 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-model-cache\") pod \"86482ac6-f516-47ca-9431-39ae80ab7ed3\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " Jul 09 23:56:36.158987 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.158826 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-home\") pod \"86482ac6-f516-47ca-9431-39ae80ab7ed3\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " Jul 09 23:56:36.158987 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.158842 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-dshm\") pod \"86482ac6-f516-47ca-9431-39ae80ab7ed3\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " Jul 09 23:56:36.158987 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.158881 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbbm8\" (UniqueName: \"kubernetes.io/projected/86482ac6-f516-47ca-9431-39ae80ab7ed3-kube-api-access-sbbm8\") pod \"86482ac6-f516-47ca-9431-39ae80ab7ed3\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " Jul 09 23:56:36.158987 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.158898 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-tmp-dir\") pod \"86482ac6-f516-47ca-9431-39ae80ab7ed3\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " Jul 09 23:56:36.158987 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.158937 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86482ac6-f516-47ca-9431-39ae80ab7ed3-tls-certs\") pod \"86482ac6-f516-47ca-9431-39ae80ab7ed3\" (UID: \"86482ac6-f516-47ca-9431-39ae80ab7ed3\") " Jul 09 23:56:36.159281 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.159114 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-home" (OuterVolumeSpecName: "home") pod "86482ac6-f516-47ca-9431-39ae80ab7ed3" (UID: "86482ac6-f516-47ca-9431-39ae80ab7ed3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:56:36.159281 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.159136 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-model-cache" (OuterVolumeSpecName: "model-cache") pod "86482ac6-f516-47ca-9431-39ae80ab7ed3" (UID: "86482ac6-f516-47ca-9431-39ae80ab7ed3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:56:36.159382 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.159312 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86482ac6-f516-47ca-9431-39ae80ab7ed3" (UID: "86482ac6-f516-47ca-9431-39ae80ab7ed3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:56:36.159382 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.159332 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-model-cache\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:56:36.159382 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.159352 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-home\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:56:36.161278 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.161245 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86482ac6-f516-47ca-9431-39ae80ab7ed3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86482ac6-f516-47ca-9431-39ae80ab7ed3" (UID: "86482ac6-f516-47ca-9431-39ae80ab7ed3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:56:36.161392 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.161346 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-dshm" (OuterVolumeSpecName: "dshm") pod "86482ac6-f516-47ca-9431-39ae80ab7ed3" (UID: "86482ac6-f516-47ca-9431-39ae80ab7ed3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:56:36.161392 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.161347 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86482ac6-f516-47ca-9431-39ae80ab7ed3-kube-api-access-sbbm8" (OuterVolumeSpecName: "kube-api-access-sbbm8") pod "86482ac6-f516-47ca-9431-39ae80ab7ed3" (UID: "86482ac6-f516-47ca-9431-39ae80ab7ed3"). InnerVolumeSpecName "kube-api-access-sbbm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:56:36.260682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.260646 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86482ac6-f516-47ca-9431-39ae80ab7ed3-tls-certs\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:56:36.260682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.260677 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-dshm\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:56:36.260682 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.260686 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbbm8\" (UniqueName: \"kubernetes.io/projected/86482ac6-f516-47ca-9431-39ae80ab7ed3-kube-api-access-sbbm8\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:56:36.260910 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.260695 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86482ac6-f516-47ca-9431-39ae80ab7ed3-tmp-dir\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 09 23:56:36.534524 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.534439 2562 generic.go:356] "Generic (PLEG): container finished" podID="86482ac6-f516-47ca-9431-39ae80ab7ed3" containerID="0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c" exitCode=0 Jul 09 23:56:36.534524 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.534494 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" Jul 09 23:56:36.534713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.534523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" event={"ID":"86482ac6-f516-47ca-9431-39ae80ab7ed3","Type":"ContainerDied","Data":"0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c"} Jul 09 23:56:36.534713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.534568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp" event={"ID":"86482ac6-f516-47ca-9431-39ae80ab7ed3","Type":"ContainerDied","Data":"3494fb02e6c31a585537389dfb20c582102e5df73845f02c625f04300c50990b"} Jul 09 23:56:36.534713 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.534586 2562 scope.go:117] "RemoveContainer" containerID="0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c" Jul 09 23:56:36.543863 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.543837 2562 scope.go:117] "RemoveContainer" containerID="0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c" Jul 09 23:56:36.544134 ip-10-0-143-88 kubenswrapper[2562]: E0709 23:56:36.544116 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c\": container with ID starting with 0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c not found: ID does not exist" containerID="0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c" Jul 09 23:56:36.544225 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.544142 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c"} err="failed to get container status \"0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c\": rpc error: code = NotFound desc = could not find container \"0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c\": container with ID starting with 0d0393f871b1ba2fee6a1abf61ab65dbe182425ad57f8ddd095ab8b09c79d04c not found: ID does not exist" Jul 09 23:56:36.554277 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.554249 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp"] Jul 09 23:56:36.559893 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.559870 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7b0dea16/e2e-lifecycle-model-7b0dea16-kserve-785f4d58dc-pxmzp"] Jul 09 23:56:36.618338 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:36.618306 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86482ac6-f516-47ca-9431-39ae80ab7ed3" path="/var/lib/kubelet/pods/86482ac6-f516-47ca-9431-39ae80ab7ed3/volumes" Jul 09 23:56:57.033499 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:57.033455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:56:57.555099 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:56:57.555064 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:00.780551 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:00.780514 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:57:03.036949 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:03.036916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:25.702778 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:25.702742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:25.800424 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:25.800385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:25.964410 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:25.964328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:26.189937 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:26.189906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:27.214272 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:27.214230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:27.313511 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:27.313479 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:27.413496 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:27.413464 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:28.107647 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:28.107611 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:28.207142 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:28.207118 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:28.307293 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:28.307260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:29.603419 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:29.603386 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:57:29.641435 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:29.641403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:35.264175 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:35.264142 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:35.327048 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:35.327012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:35.900343 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:35.900309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:41.476196 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:41.476156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:41.580864 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:41.580824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:41.679943 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:41.679912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:42.196578 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:42.196542 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:57:52.171937 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:52.171889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:52.490303 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:52.490224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:57:52.703716 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:52.703684 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:57:52.708622 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:57:52.708601 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 09 23:58:23.431563 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:23.431526 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:24.043953 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:24.043924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:24.521313 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:24.521279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:28.954235 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:28.954182 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:29.006235 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:29.006169 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:29.291600 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:29.291569 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:29.613756 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:29.613658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:29.810556 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:29.810522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:32.649377 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:32.649346 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:32.738513 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:32.738480 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:32.804640 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:32.804603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:33.034991 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:33.034960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:33.208724 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:33.208686 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:33.309501 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:33.309417 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:38.580553 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:38.580518 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:58:47.166766 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:47.166728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:47.508854 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:47.508818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:47.818577 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:47.818492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:47.984549 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:47.984517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:50.741009 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:50.740972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:50.835923 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:50.835867 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:50.905262 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:50.905232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:51.387284 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:51.387243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:51.484393 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:51.484364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:51.583317 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:51.583284 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:53.982588 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:53.982554 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:54.083190 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:54.083156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:58:54.681652 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:54.681621 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:58:55.241003 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:58:55.240972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:05.601707 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.601671 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77"] Jul 09 23:59:05.602269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.602246 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86482ac6-f516-47ca-9431-39ae80ab7ed3" containerName="main" Jul 09 23:59:05.602269 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.602268 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="86482ac6-f516-47ca-9431-39ae80ab7ed3" containerName="main" Jul 09 23:59:05.602451 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.602351 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="86482ac6-f516-47ca-9431-39ae80ab7ed3" containerName="main" Jul 09 23:59:05.605621 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.605604 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.608265 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.608239 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4e0be9\"/\"kube-root-ca.crt\"" Jul 09 23:59:05.609685 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.609662 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4e0be9\"/\"openshift-service-ca.crt\"" Jul 09 23:59:05.609797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.609691 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4e0be9\"/\"default-dockercfg-jx9tr\"" Jul 09 23:59:05.609797 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.609739 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4e0be9\"/\"test-model-4e0be9-kserve-self-signed-certs\"" Jul 09 23:59:05.614603 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.614582 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77"] Jul 09 23:59:05.629080 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.629051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:05.766783 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.766741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-model-cache\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.766783 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.766783 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-tmp-dir\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.767009 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.766807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s97h\" (UniqueName: \"kubernetes.io/projected/b56597f9-eb29-4628-b92b-c3b62660867f-kube-api-access-4s97h\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.767009 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.766923 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-dshm\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.767009 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.766961 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56597f9-eb29-4628-b92b-c3b62660867f-tls-certs\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.767009 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.766978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-home\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868106 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868022 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-model-cache\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868106 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868057 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-tmp-dir\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868106 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s97h\" (UniqueName: \"kubernetes.io/projected/b56597f9-eb29-4628-b92b-c3b62660867f-kube-api-access-4s97h\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868167 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-dshm\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868191 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56597f9-eb29-4628-b92b-c3b62660867f-tls-certs\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868369 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868229 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-home\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868515 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868494 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-tmp-dir\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868607 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-model-cache\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.868607 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.868601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-home\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.870719 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.870699 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-dshm\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.870914 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.870896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56597f9-eb29-4628-b92b-c3b62660867f-tls-certs\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.875923 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.875904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s97h\" (UniqueName: \"kubernetes.io/projected/b56597f9-eb29-4628-b92b-c3b62660867f-kube-api-access-4s97h\") pod \"test-model-4e0be9-kserve-76876f8f47-6cj77\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:05.915871 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:05.915826 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:06.051787 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:06.051764 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77"] Jul 09 23:59:06.054690 ip-10-0-143-88 kubenswrapper[2562]: W0709 23:59:06.054663 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb56597f9_eb29_4628_b92b_c3b62660867f.slice/crio-630bd97941bb7c10570aa8447ddd1233818bdb8ab85b2e88090e079f6d047112 WatchSource:0}: Error finding container 630bd97941bb7c10570aa8447ddd1233818bdb8ab85b2e88090e079f6d047112: Status 404 returned error can't find the container with id 630bd97941bb7c10570aa8447ddd1233818bdb8ab85b2e88090e079f6d047112 Jul 09 23:59:06.200285 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:06.200179 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" event={"ID":"b56597f9-eb29-4628-b92b-c3b62660867f","Type":"ContainerStarted","Data":"8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be"} Jul 09 23:59:06.200285 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:06.200245 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" event={"ID":"b56597f9-eb29-4628-b92b-c3b62660867f","Type":"ContainerStarted","Data":"630bd97941bb7c10570aa8447ddd1233818bdb8ab85b2e88090e079f6d047112"} Jul 09 23:59:06.216238 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:06.216170 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" podStartSLOduration=1.216156285 podStartE2EDuration="1.216156285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:59:06.215549753 +0000 UTC m=+1874.096301716" watchObservedRunningTime="2026-07-09 23:59:06.216156285 +0000 UTC m=+1874.096908266" Jul 09 23:59:09.685818 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:09.685778 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 09 23:59:12.177627 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:12.177553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:12.279109 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:12.279059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:15.916186 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:15.916088 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:15.916186 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:15.916147 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:15.928810 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:15.928778 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:16.259007 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:16.258973 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 09 23:59:28.478850 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:28.478799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:35.244847 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:35.244768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:43.014023 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:43.013981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:44.750736 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:44.750667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:44.797965 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:44.797935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 09 23:59:46.183391 ip-10-0-143-88 kubenswrapper[2562]: I0709 23:59:46.183356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:00:00.158043 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.158002 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p"] Jul 10 00:00:00.161674 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.161652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" Jul 10 00:00:00.164298 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.164275 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5vt95\"" Jul 10 00:00:00.179682 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.179645 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p"] Jul 10 00:00:00.195279 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.195252 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" Jul 10 00:00:00.348389 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.348362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p"] Jul 10 00:00:00.351673 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:00:00.351638 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6daa335_8527_451e_9c0d_2982f7417b86.slice/crio-e17c3496a89014dbd8b1fb8c3ff96b84aa41ec66cdeedde1f82df6b0284991ea WatchSource:0}: Error finding container e17c3496a89014dbd8b1fb8c3ff96b84aa41ec66cdeedde1f82df6b0284991ea: Status 404 returned error can't find the container with id e17c3496a89014dbd8b1fb8c3ff96b84aa41ec66cdeedde1f82df6b0284991ea Jul 10 00:00:00.353693 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.353672 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:00:00.456600 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.456516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" event={"ID":"c6daa335-8527-451e-9c0d-2982f7417b86","Type":"ContainerStarted","Data":"e17c3496a89014dbd8b1fb8c3ff96b84aa41ec66cdeedde1f82df6b0284991ea"} Jul 10 00:00:00.893736 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:00.893701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:00:08.494037 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:08.493997 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" event={"ID":"c6daa335-8527-451e-9c0d-2982f7417b86","Type":"ContainerStarted","Data":"9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d"} Jul 10 00:00:08.507652 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:08.507603 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" podStartSLOduration=1.539601855 podStartE2EDuration="8.507588685s" totalImagesPullingTime="6.96798683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:00:08.507096017 +0000 UTC m=+1936.387847976" watchObservedRunningTime="2026-07-10 00:00:08.507588685 +0000 UTC m=+1936.388340645" Jul 10 00:00:14.076352 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:14.076309 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77"] Jul 10 00:00:14.094299 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:14.094266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:15.131069 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:15.131035 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:15.286587 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:15.286527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:15.667179 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:15.667147 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:15.772958 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:15.772918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:15.869787 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:15.869747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:15.978290 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:15.978161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:20.283530 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:20.283492 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:00:20.743497 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:20.743399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:25.080828 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:25.080784 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:25.182597 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:25.182566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:29.099447 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.099395 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" podUID="b56597f9-eb29-4628-b92b-c3b62660867f" containerName="main" containerID="cri-o://8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be" gracePeriod=45 Jul 10 00:00:29.340384 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.340356 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 10 00:00:29.433531 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433431 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-home\") pod \"b56597f9-eb29-4628-b92b-c3b62660867f\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " Jul 10 00:00:29.433531 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433509 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-model-cache\") pod \"b56597f9-eb29-4628-b92b-c3b62660867f\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " Jul 10 00:00:29.433763 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433535 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-tmp-dir\") pod \"b56597f9-eb29-4628-b92b-c3b62660867f\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " Jul 10 00:00:29.433763 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433564 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s97h\" (UniqueName: \"kubernetes.io/projected/b56597f9-eb29-4628-b92b-c3b62660867f-kube-api-access-4s97h\") pod \"b56597f9-eb29-4628-b92b-c3b62660867f\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " Jul 10 00:00:29.433763 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433595 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56597f9-eb29-4628-b92b-c3b62660867f-tls-certs\") pod \"b56597f9-eb29-4628-b92b-c3b62660867f\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " Jul 10 00:00:29.433763 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433620 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-dshm\") pod \"b56597f9-eb29-4628-b92b-c3b62660867f\" (UID: \"b56597f9-eb29-4628-b92b-c3b62660867f\") " Jul 10 00:00:29.433763 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433739 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-home" (OuterVolumeSpecName: "home") pod "b56597f9-eb29-4628-b92b-c3b62660867f" (UID: "b56597f9-eb29-4628-b92b-c3b62660867f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:00:29.433976 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433767 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b56597f9-eb29-4628-b92b-c3b62660867f" (UID: "b56597f9-eb29-4628-b92b-c3b62660867f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:00:29.433976 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.433881 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b56597f9-eb29-4628-b92b-c3b62660867f" (UID: "b56597f9-eb29-4628-b92b-c3b62660867f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:00:29.434086 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.434032 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-home\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:00:29.434086 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.434045 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-model-cache\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:00:29.434086 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.434054 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-tmp-dir\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:00:29.436067 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.436036 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56597f9-eb29-4628-b92b-c3b62660867f-kube-api-access-4s97h" (OuterVolumeSpecName: "kube-api-access-4s97h") pod "b56597f9-eb29-4628-b92b-c3b62660867f" (UID: "b56597f9-eb29-4628-b92b-c3b62660867f"). InnerVolumeSpecName "kube-api-access-4s97h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:00:29.436067 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.436040 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-dshm" (OuterVolumeSpecName: "dshm") pod "b56597f9-eb29-4628-b92b-c3b62660867f" (UID: "b56597f9-eb29-4628-b92b-c3b62660867f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:00:29.436234 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.436034 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b56597f9-eb29-4628-b92b-c3b62660867f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b56597f9-eb29-4628-b92b-c3b62660867f" (UID: "b56597f9-eb29-4628-b92b-c3b62660867f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:00:29.535480 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.535438 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s97h\" (UniqueName: \"kubernetes.io/projected/b56597f9-eb29-4628-b92b-c3b62660867f-kube-api-access-4s97h\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:00:29.535480 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.535473 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b56597f9-eb29-4628-b92b-c3b62660867f-tls-certs\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:00:29.535480 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.535484 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b56597f9-eb29-4628-b92b-c3b62660867f-dshm\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:00:29.590147 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.590111 2562 generic.go:356] "Generic (PLEG): container finished" podID="b56597f9-eb29-4628-b92b-c3b62660867f" containerID="8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be" exitCode=0 Jul 10 00:00:29.590339 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.590159 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" Jul 10 00:00:29.590339 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.590191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" event={"ID":"b56597f9-eb29-4628-b92b-c3b62660867f","Type":"ContainerDied","Data":"8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be"} Jul 10 00:00:29.590339 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.590259 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77" event={"ID":"b56597f9-eb29-4628-b92b-c3b62660867f","Type":"ContainerDied","Data":"630bd97941bb7c10570aa8447ddd1233818bdb8ab85b2e88090e079f6d047112"} Jul 10 00:00:29.590339 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.590277 2562 scope.go:117] "RemoveContainer" containerID="8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be" Jul 10 00:00:29.599812 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.599796 2562 scope.go:117] "RemoveContainer" containerID="8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be" Jul 10 00:00:29.600107 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:00:29.600084 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be\": container with ID starting with 8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be not found: ID does not exist" containerID="8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be" Jul 10 00:00:29.600162 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.600116 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be"} err="failed to get container status \"8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be\": rpc error: code = NotFound desc = could not find container \"8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be\": container with ID starting with 8cf1c0ec6060f41b1709f304b7044d434a847f195e9bea9d12f60d51be6063be not found: ID does not exist" Jul 10 00:00:29.610027 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.609999 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77"] Jul 10 00:00:29.613456 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:29.613433 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4e0be9/test-model-4e0be9-kserve-76876f8f47-6cj77"] Jul 10 00:00:30.618222 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:30.618165 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56597f9-eb29-4628-b92b-c3b62660867f" path="/var/lib/kubelet/pods/b56597f9-eb29-4628-b92b-c3b62660867f/volumes" Jul 10 00:00:57.528478 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:57.528448 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:57.776535 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:57.776502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:58.001430 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.001344 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:58.172880 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.172843 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:58.582444 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.582403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:58.679468 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.679438 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:58.780775 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.780740 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:00:58.846580 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.846487 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p"] Jul 10 00:00:58.846773 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.846687 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" podUID="c6daa335-8527-451e-9c0d-2982f7417b86" containerName="cleanup" containerID="cri-o://9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d" gracePeriod=30 Jul 10 00:00:58.858364 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:00:58.858329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:01.981831 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:01.981795 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:01:04.214381 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:04.214342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:04.314285 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:04.314250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:04.582025 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:04.581992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:04.717660 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:04.717630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:05.051030 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:05.050987 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:05.346649 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:05.346572 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:05.475886 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:05.475841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:08.508886 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:08.508855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:08.600319 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:08.600288 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:08.661926 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:08.661895 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:08.762662 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:08.762572 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:09.055673 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:09.055573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:09.161004 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:09.160970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:13.004046 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:13.004011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:13.441477 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:13.441440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:14.495545 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:14.495514 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:14.548704 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:14.548672 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:16.489830 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:16.489797 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:01:16.568593 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:16.568560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:16.653059 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:16.653018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:16.729960 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:16.729927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:16.837175 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:16.837134 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:17.196013 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:17.195927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:17.297319 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:17.297282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:21.032916 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:21.032880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:29.022174 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.022127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:29.043007 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.042984 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" Jul 10 00:01:29.375086 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.375050 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:29.571734 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.571703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:29.867978 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.867950 2562 generic.go:356] "Generic (PLEG): container finished" podID="c6daa335-8527-451e-9c0d-2982f7417b86" containerID="9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d" exitCode=137 Jul 10 00:01:29.868137 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.867990 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" Jul 10 00:01:29.868137 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.868026 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" event={"ID":"c6daa335-8527-451e-9c0d-2982f7417b86","Type":"ContainerDied","Data":"9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d"} Jul 10 00:01:29.868137 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.868082 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p" event={"ID":"c6daa335-8527-451e-9c0d-2982f7417b86","Type":"ContainerDied","Data":"e17c3496a89014dbd8b1fb8c3ff96b84aa41ec66cdeedde1f82df6b0284991ea"} Jul 10 00:01:29.868137 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.868112 2562 scope.go:117] "RemoveContainer" containerID="9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d" Jul 10 00:01:29.877191 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.877174 2562 scope.go:117] "RemoveContainer" containerID="9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d" Jul 10 00:01:29.877482 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:01:29.877459 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d\": container with ID starting with 9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d not found: ID does not exist" containerID="9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d" Jul 10 00:01:29.877534 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.877491 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d"} err="failed to get container status \"9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d\": rpc error: code = NotFound desc = could not find container \"9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d\": container with ID starting with 9d21527a0d306a7f81056147663fc7a180916dd4578533c0896a3b447e0a965d not found: ID does not exist" Jul 10 00:01:29.888160 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.888123 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p"] Jul 10 00:01:29.893884 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.893860 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4f4769-29727360-t862p"] Jul 10 00:01:29.989363 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:29.989318 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:30.036782 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:30.036746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:30.617791 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:30.617758 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6daa335-8527-451e-9c0d-2982f7417b86" path="/var/lib/kubelet/pods/c6daa335-8527-451e-9c0d-2982f7417b86/volumes" Jul 10 00:01:33.983737 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:33.983708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:01:35.311972 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:35.311925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:35.412612 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:35.412575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:35.561635 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:35.561602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:35.848270 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:35.848237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:36.061030 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:36.060998 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:36.433982 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:36.433949 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:36.555666 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:36.555629 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:36.652497 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:36.652463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:36.829537 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:36.829502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:42.059442 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:42.059409 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:42.126782 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:42.126752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:42.313352 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:42.313256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:42.524960 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:42.524927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:42.849685 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:42.849639 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:43.154253 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:43.154130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:43.485889 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:43.485764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:45.741485 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:45.741440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:45.851552 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:45.851510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:45.969490 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:45.969458 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:46.089776 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:46.089745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:46.381007 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:46.380920 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:46.479133 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:46.479095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:49.377266 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:49.377178 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:01:50.218032 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:50.218000 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:50.548127 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:50.548100 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:50.840521 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:50.840434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:51.126510 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:51.126421 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:53.804729 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:53.804695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:53.928063 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:53.928025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:54.015005 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:54.014972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:54.327432 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:54.327382 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:54.797660 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:54.797625 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:54.898278 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:54.898243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:01:58.264998 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:01:58.264959 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:02:08.079886 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:08.079847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:02:08.283895 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:08.283858 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:02:08.329861 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:08.329814 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:02:16.235739 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:16.235657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:02:16.336866 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:16.336832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:02:27.886754 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:27.886721 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:02:31.284048 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:31.284010 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:02:42.904855 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:42.904821 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:02:48.778784 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:48.778746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:02:52.731017 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:52.730988 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 10 00:02:52.737729 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:52.737708 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 10 00:02:59.483997 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:02:59.483958 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:03:15.376246 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:03:15.376196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:03:30.379984 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:03:30.379951 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:03:44.687158 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:03:44.687079 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:03:59.985739 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:03:59.985701 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:04:08.638684 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:08.638649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:04:16.287937 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:16.287899 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:04:30.876947 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:30.876911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:04:31.811213 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:31.811177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:04:34.384434 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:34.384384 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:04:39.189088 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.189053 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p"] Jul 10 00:04:39.189511 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.189428 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6daa335-8527-451e-9c0d-2982f7417b86" containerName="cleanup" Jul 10 00:04:39.189511 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.189441 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6daa335-8527-451e-9c0d-2982f7417b86" containerName="cleanup" Jul 10 00:04:39.189511 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.189458 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56597f9-eb29-4628-b92b-c3b62660867f" containerName="main" Jul 10 00:04:39.189511 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.189463 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56597f9-eb29-4628-b92b-c3b62660867f" containerName="main" Jul 10 00:04:39.189642 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.189533 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56597f9-eb29-4628-b92b-c3b62660867f" containerName="main" Jul 10 00:04:39.189642 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.189543 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6daa335-8527-451e-9c0d-2982f7417b86" containerName="cleanup" Jul 10 00:04:39.192665 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.192648 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.196179 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.196151 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"openshift-service-ca.crt\"" Jul 10 00:04:39.196179 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.196160 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"default-dockercfg-98dhz\"" Jul 10 00:04:39.196179 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.196176 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"kube-root-ca.crt\"" Jul 10 00:04:39.196402 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.196277 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4c394e\"/\"sub-test-model-4c394e-kserve-self-signed-certs\"" Jul 10 00:04:39.199319 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.199298 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p"] Jul 10 00:04:39.250284 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.250251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-dshm\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.250284 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.250287 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-home\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.250520 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.250304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-model-cache\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.250520 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.250392 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tls-certs\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.250520 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.250413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26gkh\" (UniqueName: \"kubernetes.io/projected/0936b88c-3b5c-466d-ba10-9e9030e52fb6-kube-api-access-26gkh\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.250520 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.250450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tmp-dir\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.351738 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.351696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-home\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.351738 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.351738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-model-cache\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.351993 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.351820 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tls-certs\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.351993 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.351847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26gkh\" (UniqueName: \"kubernetes.io/projected/0936b88c-3b5c-466d-ba10-9e9030e52fb6-kube-api-access-26gkh\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.351993 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.351895 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tmp-dir\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.352147 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.352124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-dshm\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.352197 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.352158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-home\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.352293 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.352265 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tmp-dir\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.352354 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.352285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-model-cache\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.354356 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.354331 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-dshm\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.354468 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.354416 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tls-certs\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.359504 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.359484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26gkh\" (UniqueName: \"kubernetes.io/projected/0936b88c-3b5c-466d-ba10-9e9030e52fb6-kube-api-access-26gkh\") pod \"sub-test-model-4c394e-kserve-5f89464875-h8j6p\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.503807 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.503719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:39.638457 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.638428 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p"] Jul 10 00:04:39.639964 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:04:39.639937 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0936b88c_3b5c_466d_ba10_9e9030e52fb6.slice/crio-a61a82cc7ee32a9b8871740f09ced12a0f647afa4fdcfb1a21c362c41f1dedc7 WatchSource:0}: Error finding container a61a82cc7ee32a9b8871740f09ced12a0f647afa4fdcfb1a21c362c41f1dedc7: Status 404 returned error can't find the container with id a61a82cc7ee32a9b8871740f09ced12a0f647afa4fdcfb1a21c362c41f1dedc7 Jul 10 00:04:39.733394 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.733352 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" event={"ID":"0936b88c-3b5c-466d-ba10-9e9030e52fb6","Type":"ContainerStarted","Data":"5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326"} Jul 10 00:04:39.733687 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.733406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" event={"ID":"0936b88c-3b5c-466d-ba10-9e9030e52fb6","Type":"ContainerStarted","Data":"a61a82cc7ee32a9b8871740f09ced12a0f647afa4fdcfb1a21c362c41f1dedc7"} Jul 10 00:04:39.750518 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:39.750471 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" podStartSLOduration=0.750455672 podStartE2EDuration="750.455672ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:04:39.748654626 +0000 UTC m=+2207.629406587" watchObservedRunningTime="2026-07-10 00:04:39.750455672 +0000 UTC m=+2207.631207634" Jul 10 00:04:40.461229 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:40.461148 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:04:49.504499 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:49.504414 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:49.504499 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:49.504475 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:49.517328 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:49.517303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:49.791131 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:49.791102 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:04:51.290541 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:51.290496 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:04:55.064023 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:55.063985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:04:55.163711 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:55.163672 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:04:57.125997 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:04:57.125957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:03.287833 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:03.287788 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:05:25.881009 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:25.880921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:25.982442 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:25.982401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:26.581483 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:26.581446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:05:31.923940 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:31.923901 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p"] Jul 10 00:05:31.944408 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:31.944378 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:32.904755 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:32.904719 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:45.582809 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:45.582774 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:05:46.945150 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:46.945083 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" podUID="0936b88c-3b5c-466d-ba10-9e9030e52fb6" containerName="main" containerID="cri-o://5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326" gracePeriod=45 Jul 10 00:05:47.198696 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.198637 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:05:47.209664 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.209633 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:47.267853 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.267822 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-dshm\") pod \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " Jul 10 00:05:47.268034 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.267866 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-home\") pod \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " Jul 10 00:05:47.268034 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.267883 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-model-cache\") pod \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " Jul 10 00:05:47.268034 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.267951 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26gkh\" (UniqueName: \"kubernetes.io/projected/0936b88c-3b5c-466d-ba10-9e9030e52fb6-kube-api-access-26gkh\") pod \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " Jul 10 00:05:47.268034 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.267967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tmp-dir\") pod \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " Jul 10 00:05:47.268034 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.268004 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tls-certs\") pod \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\" (UID: \"0936b88c-3b5c-466d-ba10-9e9030e52fb6\") " Jul 10 00:05:47.268333 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.268165 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-home" (OuterVolumeSpecName: "home") pod "0936b88c-3b5c-466d-ba10-9e9030e52fb6" (UID: "0936b88c-3b5c-466d-ba10-9e9030e52fb6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:05:47.268333 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.268181 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0936b88c-3b5c-466d-ba10-9e9030e52fb6" (UID: "0936b88c-3b5c-466d-ba10-9e9030e52fb6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:05:47.268333 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.268252 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0936b88c-3b5c-466d-ba10-9e9030e52fb6" (UID: "0936b88c-3b5c-466d-ba10-9e9030e52fb6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:05:47.268333 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.268303 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-home\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:05:47.268333 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.268317 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-model-cache\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:05:47.270304 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.270275 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-dshm" (OuterVolumeSpecName: "dshm") pod "0936b88c-3b5c-466d-ba10-9e9030e52fb6" (UID: "0936b88c-3b5c-466d-ba10-9e9030e52fb6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:05:47.270425 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.270355 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0936b88c-3b5c-466d-ba10-9e9030e52fb6" (UID: "0936b88c-3b5c-466d-ba10-9e9030e52fb6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:05:47.270425 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.270391 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0936b88c-3b5c-466d-ba10-9e9030e52fb6-kube-api-access-26gkh" (OuterVolumeSpecName: "kube-api-access-26gkh") pod "0936b88c-3b5c-466d-ba10-9e9030e52fb6" (UID: "0936b88c-3b5c-466d-ba10-9e9030e52fb6"). InnerVolumeSpecName "kube-api-access-26gkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:05:47.305184 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.305151 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:47.369271 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.369232 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tls-certs\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:05:47.369271 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.369261 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-dshm\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:05:47.369271 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.369270 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26gkh\" (UniqueName: \"kubernetes.io/projected/0936b88c-3b5c-466d-ba10-9e9030e52fb6-kube-api-access-26gkh\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:05:47.369271 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:47.369278 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0936b88c-3b5c-466d-ba10-9e9030e52fb6-tmp-dir\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:05:48.059962 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.059928 2562 generic.go:356] "Generic (PLEG): container finished" podID="0936b88c-3b5c-466d-ba10-9e9030e52fb6" containerID="5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326" exitCode=0 Jul 10 00:05:48.060474 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.059970 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" Jul 10 00:05:48.060474 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.060015 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" event={"ID":"0936b88c-3b5c-466d-ba10-9e9030e52fb6","Type":"ContainerDied","Data":"5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326"} Jul 10 00:05:48.060474 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.060061 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p" event={"ID":"0936b88c-3b5c-466d-ba10-9e9030e52fb6","Type":"ContainerDied","Data":"a61a82cc7ee32a9b8871740f09ced12a0f647afa4fdcfb1a21c362c41f1dedc7"} Jul 10 00:05:48.060474 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.060088 2562 scope.go:117] "RemoveContainer" containerID="5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326" Jul 10 00:05:48.069391 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.069376 2562 scope.go:117] "RemoveContainer" containerID="5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326" Jul 10 00:05:48.069600 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:05:48.069585 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326\": container with ID starting with 5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326 not found: ID does not exist" containerID="5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326" Jul 10 00:05:48.069646 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.069607 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326"} err="failed to get container status \"5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326\": rpc error: code = NotFound desc = could not find container \"5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326\": container with ID starting with 5c12fe40966e7e6ce3c1572ff4a65ff541a38f349c96226bc590302904702326 not found: ID does not exist" Jul 10 00:05:48.080510 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.080485 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p"] Jul 10 00:05:48.083394 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.083375 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4c394e/sub-test-model-4c394e-kserve-5f89464875-h8j6p"] Jul 10 00:05:48.617578 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:48.617536 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0936b88c-3b5c-466d-ba10-9e9030e52fb6" path="/var/lib/kubelet/pods/0936b88c-3b5c-466d-ba10-9e9030e52fb6/volumes" Jul 10 00:05:50.916943 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:50.916902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:05:58.778539 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:05:58.778502 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:06:03.464847 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:03.464809 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:06:10.678966 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:10.678929 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:06:32.484112 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:32.484078 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:06:47.780414 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:47.780330 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:06:54.196166 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:54.196137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:06:55.245525 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:55.245463 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:06:56.108897 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.108860 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7224-140222187935280"] Jul 10 00:06:56.109229 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.109217 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0936b88c-3b5c-466d-ba10-9e9030e52fb6" containerName="main" Jul 10 00:06:56.109293 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.109231 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0936b88c-3b5c-466d-ba10-9e9030e52fb6" containerName="main" Jul 10 00:06:56.109337 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.109296 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0936b88c-3b5c-466d-ba10-9e9030e52fb6" containerName="main" Jul 10 00:06:56.112366 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.112342 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187935280" Jul 10 00:06:56.114647 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.114621 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-c9bdn\"" Jul 10 00:06:56.118349 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.118328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222187935280"] Jul 10 00:06:56.207947 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.207910 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkm49\" (UniqueName: \"kubernetes.io/projected/f4669ebc-5279-440a-bfc7-bf5819ebb542-kube-api-access-nkm49\") pod \"test-curl-7224-140222187935280\" (UID: \"f4669ebc-5279-440a-bfc7-bf5819ebb542\") " pod="openshift-ingress/test-curl-7224-140222187935280" Jul 10 00:06:56.308396 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.308358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkm49\" (UniqueName: \"kubernetes.io/projected/f4669ebc-5279-440a-bfc7-bf5819ebb542-kube-api-access-nkm49\") pod \"test-curl-7224-140222187935280\" (UID: \"f4669ebc-5279-440a-bfc7-bf5819ebb542\") " pod="openshift-ingress/test-curl-7224-140222187935280" Jul 10 00:06:56.315980 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.315956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkm49\" (UniqueName: \"kubernetes.io/projected/f4669ebc-5279-440a-bfc7-bf5819ebb542-kube-api-access-nkm49\") pod \"test-curl-7224-140222187935280\" (UID: \"f4669ebc-5279-440a-bfc7-bf5819ebb542\") " pod="openshift-ingress/test-curl-7224-140222187935280" Jul 10 00:06:56.422951 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.422871 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187935280" Jul 10 00:06:56.555957 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.555930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222187935280"] Jul 10 00:06:56.558193 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:06:56.558169 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4669ebc_5279_440a_bfc7_bf5819ebb542.slice/crio-f07106a14bf7448831db86c7b4d99dab0d9d77fe6e30171e2d7071acd8f42052 WatchSource:0}: Error finding container f07106a14bf7448831db86c7b4d99dab0d9d77fe6e30171e2d7071acd8f42052: Status 404 returned error can't find the container with id f07106a14bf7448831db86c7b4d99dab0d9d77fe6e30171e2d7071acd8f42052 Jul 10 00:06:56.559451 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:56.559432 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:06:57.382694 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:57.382669 2562 generic.go:356] "Generic (PLEG): container finished" podID="f4669ebc-5279-440a-bfc7-bf5819ebb542" containerID="aafa21ad2811ca2be10f6fbb6524362cd1460cc7a228e9e38ff4c1438a2423ea" exitCode=0 Jul 10 00:06:57.383026 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:57.382748 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222187935280" event={"ID":"f4669ebc-5279-440a-bfc7-bf5819ebb542","Type":"ContainerDied","Data":"aafa21ad2811ca2be10f6fbb6524362cd1460cc7a228e9e38ff4c1438a2423ea"} Jul 10 00:06:57.383026 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:57.382793 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222187935280" event={"ID":"f4669ebc-5279-440a-bfc7-bf5819ebb542","Type":"ContainerStarted","Data":"f07106a14bf7448831db86c7b4d99dab0d9d77fe6e30171e2d7071acd8f42052"} Jul 10 00:06:58.527561 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.527537 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187935280" Jul 10 00:06:58.545270 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.545246 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7224-140222187935280_f4669ebc-5279-440a-bfc7-bf5819ebb542/test-curl-7224-140222187935280/0.log" Jul 10 00:06:58.568036 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.568004 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7224-140222187935280"] Jul 10 00:06:58.573309 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.573285 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7224-140222187935280"] Jul 10 00:06:58.627509 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.627471 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkm49\" (UniqueName: \"kubernetes.io/projected/f4669ebc-5279-440a-bfc7-bf5819ebb542-kube-api-access-nkm49\") pod \"f4669ebc-5279-440a-bfc7-bf5819ebb542\" (UID: \"f4669ebc-5279-440a-bfc7-bf5819ebb542\") " Jul 10 00:06:58.629867 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.629830 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4669ebc-5279-440a-bfc7-bf5819ebb542-kube-api-access-nkm49" (OuterVolumeSpecName: "kube-api-access-nkm49") pod "f4669ebc-5279-440a-bfc7-bf5819ebb542" (UID: "f4669ebc-5279-440a-bfc7-bf5819ebb542"). InnerVolumeSpecName "kube-api-access-nkm49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:06:58.686258 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.686171 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7224-140222187936304"] Jul 10 00:06:58.686603 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.686589 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4669ebc-5279-440a-bfc7-bf5819ebb542" containerName="test-curl-7224-140222187935280" Jul 10 00:06:58.686654 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.686605 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4669ebc-5279-440a-bfc7-bf5819ebb542" containerName="test-curl-7224-140222187935280" Jul 10 00:06:58.686687 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.686680 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4669ebc-5279-440a-bfc7-bf5819ebb542" containerName="test-curl-7224-140222187935280" Jul 10 00:06:58.689537 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.689522 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:06:58.696473 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.696451 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222187936304"] Jul 10 00:06:58.728402 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.728373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56nl\" (UniqueName: \"kubernetes.io/projected/63177177-08f4-4c21-9739-f352066d6805-kube-api-access-d56nl\") pod \"test-curl-7224-140222187936304\" (UID: \"63177177-08f4-4c21-9739-f352066d6805\") " pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:06:58.728560 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.728473 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkm49\" (UniqueName: \"kubernetes.io/projected/f4669ebc-5279-440a-bfc7-bf5819ebb542-kube-api-access-nkm49\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:06:58.829788 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.829752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56nl\" (UniqueName: \"kubernetes.io/projected/63177177-08f4-4c21-9739-f352066d6805-kube-api-access-d56nl\") pod \"test-curl-7224-140222187936304\" (UID: \"63177177-08f4-4c21-9739-f352066d6805\") " pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:06:58.837384 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.837353 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56nl\" (UniqueName: \"kubernetes.io/projected/63177177-08f4-4c21-9739-f352066d6805-kube-api-access-d56nl\") pod \"test-curl-7224-140222187936304\" (UID: \"63177177-08f4-4c21-9739-f352066d6805\") " pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:06:58.998856 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:58.998770 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:06:59.129630 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:59.129604 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222187936304"] Jul 10 00:06:59.131670 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:06:59.131635 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63177177_08f4_4c21_9739_f352066d6805.slice/crio-0c001b4c46162eba83d48528d6b21d9e7a4bbdbff534624d81781ae98e5e7fa0 WatchSource:0}: Error finding container 0c001b4c46162eba83d48528d6b21d9e7a4bbdbff534624d81781ae98e5e7fa0: Status 404 returned error can't find the container with id 0c001b4c46162eba83d48528d6b21d9e7a4bbdbff534624d81781ae98e5e7fa0 Jul 10 00:06:59.392483 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:59.392448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222187936304" event={"ID":"63177177-08f4-4c21-9739-f352066d6805","Type":"ContainerStarted","Data":"0c001b4c46162eba83d48528d6b21d9e7a4bbdbff534624d81781ae98e5e7fa0"} Jul 10 00:06:59.393760 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:59.393734 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187935280" Jul 10 00:06:59.393760 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:06:59.393754 2562 scope.go:117] "RemoveContainer" containerID="aafa21ad2811ca2be10f6fbb6524362cd1460cc7a228e9e38ff4c1438a2423ea" Jul 10 00:06:59.610117 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:06:59.610085 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63177177_08f4_4c21_9739_f352066d6805.slice/crio-7beb7e04081894149a301b46efea0b2a98469013ec52ef3d84eb7bfcd6de33c1.scope\": RecentStats: unable to find data in memory cache]" Jul 10 00:07:00.400311 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:00.400277 2562 generic.go:356] "Generic (PLEG): container finished" podID="63177177-08f4-4c21-9739-f352066d6805" containerID="7beb7e04081894149a301b46efea0b2a98469013ec52ef3d84eb7bfcd6de33c1" exitCode=0 Jul 10 00:07:00.400511 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:00.400367 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222187936304" event={"ID":"63177177-08f4-4c21-9739-f352066d6805","Type":"ContainerDied","Data":"7beb7e04081894149a301b46efea0b2a98469013ec52ef3d84eb7bfcd6de33c1"} Jul 10 00:07:00.617642 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:00.617611 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4669ebc-5279-440a-bfc7-bf5819ebb542" path="/var/lib/kubelet/pods/f4669ebc-5279-440a-bfc7-bf5819ebb542/volumes" Jul 10 00:07:01.548610 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.548582 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:07:01.569525 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.569496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7224-140222187936304_63177177-08f4-4c21-9739-f352066d6805/test-curl-7224-140222187936304/0.log" Jul 10 00:07:01.592376 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.592344 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7224-140222187936304"] Jul 10 00:07:01.597137 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.597114 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7224-140222187936304"] Jul 10 00:07:01.652676 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.652642 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d56nl\" (UniqueName: \"kubernetes.io/projected/63177177-08f4-4c21-9739-f352066d6805-kube-api-access-d56nl\") pod \"63177177-08f4-4c21-9739-f352066d6805\" (UID: \"63177177-08f4-4c21-9739-f352066d6805\") " Jul 10 00:07:01.654829 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.654807 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63177177-08f4-4c21-9739-f352066d6805-kube-api-access-d56nl" (OuterVolumeSpecName: "kube-api-access-d56nl") pod "63177177-08f4-4c21-9739-f352066d6805" (UID: "63177177-08f4-4c21-9739-f352066d6805"). InnerVolumeSpecName "kube-api-access-d56nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:07:01.753464 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.753429 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d56nl\" (UniqueName: \"kubernetes.io/projected/63177177-08f4-4c21-9739-f352066d6805-kube-api-access-d56nl\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:07:01.762679 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.762649 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:01.763012 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.763000 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63177177-08f4-4c21-9739-f352066d6805" containerName="test-curl-7224-140222187936304" Jul 10 00:07:01.763058 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.763014 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="63177177-08f4-4c21-9739-f352066d6805" containerName="test-curl-7224-140222187936304" Jul 10 00:07:01.763094 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.763072 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="63177177-08f4-4c21-9739-f352066d6805" containerName="test-curl-7224-140222187936304" Jul 10 00:07:01.766028 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.766011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:01.771974 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.771944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:01.854791 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.854752 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcgd\" (UniqueName: \"kubernetes.io/projected/2f3c647d-1643-4988-a27d-62f67d1854bb-kube-api-access-qgcgd\") pod \"test-curl-7224-140222186907520\" (UID: \"2f3c647d-1643-4988-a27d-62f67d1854bb\") " pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:01.955788 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.955698 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcgd\" (UniqueName: \"kubernetes.io/projected/2f3c647d-1643-4988-a27d-62f67d1854bb-kube-api-access-qgcgd\") pod \"test-curl-7224-140222186907520\" (UID: \"2f3c647d-1643-4988-a27d-62f67d1854bb\") " pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:01.963308 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:01.963282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcgd\" (UniqueName: \"kubernetes.io/projected/2f3c647d-1643-4988-a27d-62f67d1854bb-kube-api-access-qgcgd\") pod \"test-curl-7224-140222186907520\" (UID: \"2f3c647d-1643-4988-a27d-62f67d1854bb\") " pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:02.075548 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:02.075496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:02.203051 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:02.203022 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:02.206909 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:07:02.206845 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f3c647d_1643_4988_a27d_62f67d1854bb.slice/crio-912a057df7640ec2186a903209c073c99da8a505f275896a0f9458ece40bcb75 WatchSource:0}: Error finding container 912a057df7640ec2186a903209c073c99da8a505f275896a0f9458ece40bcb75: Status 404 returned error can't find the container with id 912a057df7640ec2186a903209c073c99da8a505f275896a0f9458ece40bcb75 Jul 10 00:07:02.412243 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:02.412195 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:07:02.412243 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:02.412226 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c001b4c46162eba83d48528d6b21d9e7a4bbdbff534624d81781ae98e5e7fa0" Jul 10 00:07:02.413351 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:02.413325 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186907520" event={"ID":"2f3c647d-1643-4988-a27d-62f67d1854bb","Type":"ContainerStarted","Data":"912a057df7640ec2186a903209c073c99da8a505f275896a0f9458ece40bcb75"} Jul 10 00:07:02.427410 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:07:02.427384 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7224-140222187936304\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="63177177-08f4-4c21-9739-f352066d6805" pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:07:02.481281 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:02.481182 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:07:02.616364 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:07:02.616332 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7224-140222187936304\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="63177177-08f4-4c21-9739-f352066d6805" pod="openshift-ingress/test-curl-7224-140222187936304" Jul 10 00:07:02.618236 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:02.618215 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63177177-08f4-4c21-9739-f352066d6805" path="/var/lib/kubelet/pods/63177177-08f4-4c21-9739-f352066d6805/volumes" Jul 10 00:07:03.419721 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:03.419687 2562 generic.go:356] "Generic (PLEG): container finished" podID="2f3c647d-1643-4988-a27d-62f67d1854bb" containerID="4129ea54e934b5ebaa1ba9e9c5d495db9514c226bf8fcc1438b85e9ac60ebd37" exitCode=0 Jul 10 00:07:03.420127 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:03.419752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186907520" event={"ID":"2f3c647d-1643-4988-a27d-62f67d1854bb","Type":"ContainerDied","Data":"4129ea54e934b5ebaa1ba9e9c5d495db9514c226bf8fcc1438b85e9ac60ebd37"} Jul 10 00:07:04.566933 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.566910 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:04.586023 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.585982 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7224-140222186907520_2f3c647d-1643-4988-a27d-62f67d1854bb/test-curl-7224-140222186907520/0.log" Jul 10 00:07:04.608109 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.608079 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:04.610047 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.610023 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:04.681578 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.681492 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgcgd\" (UniqueName: \"kubernetes.io/projected/2f3c647d-1643-4988-a27d-62f67d1854bb-kube-api-access-qgcgd\") pod \"2f3c647d-1643-4988-a27d-62f67d1854bb\" (UID: \"2f3c647d-1643-4988-a27d-62f67d1854bb\") " Jul 10 00:07:04.683688 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.683664 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f3c647d-1643-4988-a27d-62f67d1854bb-kube-api-access-qgcgd" (OuterVolumeSpecName: "kube-api-access-qgcgd") pod "2f3c647d-1643-4988-a27d-62f67d1854bb" (UID: "2f3c647d-1643-4988-a27d-62f67d1854bb"). InnerVolumeSpecName "kube-api-access-qgcgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:07:04.724390 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.724363 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:04.724718 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.724705 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3c647d-1643-4988-a27d-62f67d1854bb" containerName="test-curl-7224-140222186907520" Jul 10 00:07:04.724772 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.724719 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3c647d-1643-4988-a27d-62f67d1854bb" containerName="test-curl-7224-140222186907520" Jul 10 00:07:04.724813 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.724789 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3c647d-1643-4988-a27d-62f67d1854bb" containerName="test-curl-7224-140222186907520" Jul 10 00:07:04.727756 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.727737 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:04.733120 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.733093 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:04.782922 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.782880 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcw7v\" (UniqueName: \"kubernetes.io/projected/7a04e9d5-1461-4d82-a940-3361c9841d22-kube-api-access-vcw7v\") pod \"test-curl-7224-140222186907520\" (UID: \"7a04e9d5-1461-4d82-a940-3361c9841d22\") " pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:04.783109 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.783031 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcgd\" (UniqueName: \"kubernetes.io/projected/2f3c647d-1643-4988-a27d-62f67d1854bb-kube-api-access-qgcgd\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:07:04.884245 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.884182 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcw7v\" (UniqueName: \"kubernetes.io/projected/7a04e9d5-1461-4d82-a940-3361c9841d22-kube-api-access-vcw7v\") pod \"test-curl-7224-140222186907520\" (UID: \"7a04e9d5-1461-4d82-a940-3361c9841d22\") " pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:04.891930 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:04.891893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcw7v\" (UniqueName: \"kubernetes.io/projected/7a04e9d5-1461-4d82-a940-3361c9841d22-kube-api-access-vcw7v\") pod \"test-curl-7224-140222186907520\" (UID: \"7a04e9d5-1461-4d82-a940-3361c9841d22\") " pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:05.038109 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:05.038077 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:05.170447 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:05.170415 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:05.173691 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:07:05.173658 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a04e9d5_1461_4d82_a940_3361c9841d22.slice/crio-3159b27260463c8d81f3d49ba9dc11ada829e5f86de22f0f578a8028415716d9 WatchSource:0}: Error finding container 3159b27260463c8d81f3d49ba9dc11ada829e5f86de22f0f578a8028415716d9: Status 404 returned error can't find the container with id 3159b27260463c8d81f3d49ba9dc11ada829e5f86de22f0f578a8028415716d9 Jul 10 00:07:05.429562 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:05.429459 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186907520" event={"ID":"7a04e9d5-1461-4d82-a940-3361c9841d22","Type":"ContainerStarted","Data":"3159b27260463c8d81f3d49ba9dc11ada829e5f86de22f0f578a8028415716d9"} Jul 10 00:07:05.430666 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:05.430651 2562 scope.go:117] "RemoveContainer" containerID="4129ea54e934b5ebaa1ba9e9c5d495db9514c226bf8fcc1438b85e9ac60ebd37" Jul 10 00:07:05.430737 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:05.430662 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:06.437112 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:06.437081 2562 generic.go:356] "Generic (PLEG): container finished" podID="7a04e9d5-1461-4d82-a940-3361c9841d22" containerID="8ade9a82c6a12ade2738d9c7c764f3771d246efd59bee06ac5295fe6f450878b" exitCode=0 Jul 10 00:07:06.437528 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:06.437181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186907520" event={"ID":"7a04e9d5-1461-4d82-a940-3361c9841d22","Type":"ContainerDied","Data":"8ade9a82c6a12ade2738d9c7c764f3771d246efd59bee06ac5295fe6f450878b"} Jul 10 00:07:06.617723 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:06.617691 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f3c647d-1643-4988-a27d-62f67d1854bb" path="/var/lib/kubelet/pods/2f3c647d-1643-4988-a27d-62f67d1854bb/volumes" Jul 10 00:07:07.588405 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.588381 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:07.606805 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.606776 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7224-140222186907520_7a04e9d5-1461-4d82-a940-3361c9841d22/test-curl-7224-140222186907520/0.log" Jul 10 00:07:07.631253 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.631220 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:07.634162 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.634139 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7224-140222186907520"] Jul 10 00:07:07.710997 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.710909 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcw7v\" (UniqueName: \"kubernetes.io/projected/7a04e9d5-1461-4d82-a940-3361c9841d22-kube-api-access-vcw7v\") pod \"7a04e9d5-1461-4d82-a940-3361c9841d22\" (UID: \"7a04e9d5-1461-4d82-a940-3361c9841d22\") " Jul 10 00:07:07.713144 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.713114 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a04e9d5-1461-4d82-a940-3361c9841d22-kube-api-access-vcw7v" (OuterVolumeSpecName: "kube-api-access-vcw7v") pod "7a04e9d5-1461-4d82-a940-3361c9841d22" (UID: "7a04e9d5-1461-4d82-a940-3361c9841d22"). InnerVolumeSpecName "kube-api-access-vcw7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:07:07.750778 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.750745 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"] Jul 10 00:07:07.751158 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.751145 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a04e9d5-1461-4d82-a940-3361c9841d22" containerName="test-curl-7224-140222186907520" Jul 10 00:07:07.751214 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.751161 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a04e9d5-1461-4d82-a940-3361c9841d22" containerName="test-curl-7224-140222186907520" Jul 10 00:07:07.751252 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.751243 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a04e9d5-1461-4d82-a940-3361c9841d22" containerName="test-curl-7224-140222186907520" Jul 10 00:07:07.754257 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.754234 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:07.757996 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.757973 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"] Jul 10 00:07:07.812634 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.812597 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgg69\" (UniqueName: \"kubernetes.io/projected/45096988-8878-4c1c-82b4-09af56884749-kube-api-access-mgg69\") pod \"test-curl-7224-140222186908528\" (UID: \"45096988-8878-4c1c-82b4-09af56884749\") " pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:07.812793 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.812654 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcw7v\" (UniqueName: \"kubernetes.io/projected/7a04e9d5-1461-4d82-a940-3361c9841d22-kube-api-access-vcw7v\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:07:07.913976 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.913920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgg69\" (UniqueName: \"kubernetes.io/projected/45096988-8878-4c1c-82b4-09af56884749-kube-api-access-mgg69\") pod \"test-curl-7224-140222186908528\" (UID: \"45096988-8878-4c1c-82b4-09af56884749\") " pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:07.921528 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:07.921503 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgg69\" (UniqueName: \"kubernetes.io/projected/45096988-8878-4c1c-82b4-09af56884749-kube-api-access-mgg69\") pod \"test-curl-7224-140222186908528\" (UID: \"45096988-8878-4c1c-82b4-09af56884749\") " pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:08.065522 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.065485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:08.194786 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.194760 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"] Jul 10 00:07:08.197430 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:07:08.197402 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45096988_8878_4c1c_82b4_09af56884749.slice/crio-d343abdc5939b809574da82a97e68e703efcec0b519ab7c387c2d726d2613c96 WatchSource:0}: Error finding container d343abdc5939b809574da82a97e68e703efcec0b519ab7c387c2d726d2613c96: Status 404 returned error can't find the container with id d343abdc5939b809574da82a97e68e703efcec0b519ab7c387c2d726d2613c96 Jul 10 00:07:08.454668 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.454579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186908528" event={"ID":"45096988-8878-4c1c-82b4-09af56884749","Type":"ContainerStarted","Data":"d343abdc5939b809574da82a97e68e703efcec0b519ab7c387c2d726d2613c96"} Jul 10 00:07:08.455819 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.455801 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3159b27260463c8d81f3d49ba9dc11ada829e5f86de22f0f578a8028415716d9" Jul 10 00:07:08.455819 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.455801 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:08.466327 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:07:08.466299 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7224-140222186907520\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="7a04e9d5-1461-4d82-a940-3361c9841d22" pod="openshift-ingress/test-curl-7224-140222186907520" Jul 10 00:07:08.618317 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.618283 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a04e9d5-1461-4d82-a940-3361c9841d22" path="/var/lib/kubelet/pods/7a04e9d5-1461-4d82-a940-3361c9841d22/volumes" Jul 10 00:07:08.814453 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.814414 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:08.915174 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:08.915139 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:09.462137 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:09.462100 2562 generic.go:356] "Generic (PLEG): container finished" podID="45096988-8878-4c1c-82b4-09af56884749" containerID="3af01ab97d36c902147fa52d684e3730bba72c9203ffde4facbd438bd5c506ed" exitCode=0 Jul 10 00:07:09.462339 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:09.462185 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7224-140222186908528" event={"ID":"45096988-8878-4c1c-82b4-09af56884749","Type":"ContainerDied","Data":"3af01ab97d36c902147fa52d684e3730bba72c9203ffde4facbd438bd5c506ed"} Jul 10 00:07:10.634095 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:10.634068 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:10.651772 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:10.651724 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7224-140222186908528_45096988-8878-4c1c-82b4-09af56884749/test-curl-7224-140222186908528/0.log" Jul 10 00:07:10.674047 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:10.674015 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"] Jul 10 00:07:10.676584 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:10.676562 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7224-140222186908528"] Jul 10 00:07:10.741245 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:10.741142 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgg69\" (UniqueName: \"kubernetes.io/projected/45096988-8878-4c1c-82b4-09af56884749-kube-api-access-mgg69\") pod \"45096988-8878-4c1c-82b4-09af56884749\" (UID: \"45096988-8878-4c1c-82b4-09af56884749\") " Jul 10 00:07:10.743387 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:10.743364 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45096988-8878-4c1c-82b4-09af56884749-kube-api-access-mgg69" (OuterVolumeSpecName: "kube-api-access-mgg69") pod "45096988-8878-4c1c-82b4-09af56884749" (UID: "45096988-8878-4c1c-82b4-09af56884749"). InnerVolumeSpecName "kube-api-access-mgg69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:07:10.842646 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:10.842608 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgg69\" (UniqueName: \"kubernetes.io/projected/45096988-8878-4c1c-82b4-09af56884749-kube-api-access-mgg69\") on node \"ip-10-0-143-88.ec2.internal\" DevicePath \"\"" Jul 10 00:07:11.472555 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:11.472529 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d343abdc5939b809574da82a97e68e703efcec0b519ab7c387c2d726d2613c96" Jul 10 00:07:11.472555 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:11.472546 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:11.482153 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:07:11.482128 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7224-140222186908528\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="45096988-8878-4c1c-82b4-09af56884749" pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:12.620186 ip-10-0-143-88 kubenswrapper[2562]: E0710 00:07:12.620138 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7224-140222186908528\" is forbidden: User \"system:node:ip-10-0-143-88.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-143-88.ec2.internal' and this object" podUID="45096988-8878-4c1c-82b4-09af56884749" pod="openshift-ingress/test-curl-7224-140222186908528" Jul 10 00:07:12.622453 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:12.622432 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45096988-8878-4c1c-82b4-09af56884749" path="/var/lib/kubelet/pods/45096988-8878-4c1c-82b4-09af56884749/volumes" Jul 10 00:07:14.112512 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:14.112478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:14.477036 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:14.476947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:14.577129 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:14.577088 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:17.096178 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:17.096145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:17.197140 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:17.197097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:17.977756 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:17.977721 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:07:20.202790 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:20.202754 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:20.259887 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:20.259847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:20.450315 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:20.450264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:20.681822 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:20.681790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:20.825595 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:20.825556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:21.215025 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:21.214992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:21.316508 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:21.316481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:23.349352 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:23.349320 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:27.060926 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:27.060886 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:27.159921 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:27.159892 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:27.306872 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:27.306839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:27.530386 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:27.530358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:30.380513 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:30.380472 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q7r2l"] Jul 10 00:07:33.386818 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:33.386787 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-qbnb6_7cfa21a5-a002-4495-9d8c-189edb2e0847/manager/0.log" Jul 10 00:07:33.398170 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:33.398149 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-qbnb6_7cfa21a5-a002-4495-9d8c-189edb2e0847/copy-manifests/0.log" Jul 10 00:07:35.284303 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:35.284277 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7tbp5_b516a848-78cf-4d78-850e-cd42b69fbca4/manager/0.log" Jul 10 00:07:35.387995 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:35.387964 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-c4s2h_3303cbae-7fa3-4d62-bf8a-6c8720231294/kuadrant-console-plugin/0.log" Jul 10 00:07:35.731168 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:35.731080 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q7r2l_cedcc116-847f-4f50-b3ed-182fc7e813ff/limitador/0.log" Jul 10 00:07:36.507161 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:36.507134 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-96c2g_fc7a0613-0f47-48e0-8ccf-608fc7c0f535/kube-auth-proxy/0.log" Jul 10 00:07:37.285764 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:37.285736 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tdhp9_405f9927-c151-495d-8fe1-afc009a8e57f/main/0.log" Jul 10 00:07:37.499848 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:37.499818 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-nqhjk_ba82c6f2-11b0-4b41-a18a-7d50d7314000/main/0.log" Jul 10 00:07:37.710819 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:37.710746 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vpp9w_5a9c2964-a3c7-4855-a56f-c0d2e27845bc/main/0.log" Jul 10 00:07:41.417959 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:41.417926 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:41.489315 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:41.489278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" limit=1024 Jul 10 00:07:44.006440 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:44.006408 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8gm96_b38ad682-bd50-4725-b3c1-821e12e3e15f/global-pull-secret-syncer/0.log" Jul 10 00:07:44.134599 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:44.134565 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8vqmv_b040f8e8-3927-4005-a787-52003adb5e78/konnectivity-agent/0.log" Jul 10 00:07:44.256861 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:44.256837 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-88.ec2.internal_68d79ca2f41a1fb714c439c75beb926d/haproxy/0.log" Jul 10 00:07:49.635573 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:49.635491 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7tbp5_b516a848-78cf-4d78-850e-cd42b69fbca4/manager/0.log" Jul 10 00:07:49.670535 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:49.670508 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-c4s2h_3303cbae-7fa3-4d62-bf8a-6c8720231294/kuadrant-console-plugin/0.log" Jul 10 00:07:49.859025 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:49.858992 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q7r2l_cedcc116-847f-4f50-b3ed-182fc7e813ff/limitador/0.log" Jul 10 00:07:51.272560 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.272529 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d8cffea-f5b7-47dc-931f-9b7acfbe1b32/alertmanager/0.log" Jul 10 00:07:51.299074 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.299044 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d8cffea-f5b7-47dc-931f-9b7acfbe1b32/config-reloader/0.log" Jul 10 00:07:51.320459 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.320411 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d8cffea-f5b7-47dc-931f-9b7acfbe1b32/kube-rbac-proxy-web/0.log" Jul 10 00:07:51.339955 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.339933 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d8cffea-f5b7-47dc-931f-9b7acfbe1b32/kube-rbac-proxy/0.log" Jul 10 00:07:51.363787 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.363761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d8cffea-f5b7-47dc-931f-9b7acfbe1b32/kube-rbac-proxy-metric/0.log" Jul 10 00:07:51.382898 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.382878 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d8cffea-f5b7-47dc-931f-9b7acfbe1b32/prom-label-proxy/0.log" Jul 10 00:07:51.407104 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.407080 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4d8cffea-f5b7-47dc-931f-9b7acfbe1b32/init-config-reloader/0.log" Jul 10 00:07:51.689829 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.689729 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4mks_1d7f8352-3225-4ae2-9596-edfcee60dda0/node-exporter/0.log" Jul 10 00:07:51.712392 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.712338 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4mks_1d7f8352-3225-4ae2-9596-edfcee60dda0/kube-rbac-proxy/0.log" Jul 10 00:07:51.739045 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.739019 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q4mks_1d7f8352-3225-4ae2-9596-edfcee60dda0/init-textfile/0.log" Jul 10 00:07:51.958389 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.958317 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f3bf99cc-4504-49db-ab8b-30ced52b715a/prometheus/0.log" Jul 10 00:07:51.978019 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.977986 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f3bf99cc-4504-49db-ab8b-30ced52b715a/config-reloader/0.log" Jul 10 00:07:51.998953 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:51.998921 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f3bf99cc-4504-49db-ab8b-30ced52b715a/thanos-sidecar/0.log" Jul 10 00:07:52.021331 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.021301 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f3bf99cc-4504-49db-ab8b-30ced52b715a/kube-rbac-proxy-web/0.log" Jul 10 00:07:52.041882 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.041857 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f3bf99cc-4504-49db-ab8b-30ced52b715a/kube-rbac-proxy/0.log" Jul 10 00:07:52.064354 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.064316 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f3bf99cc-4504-49db-ab8b-30ced52b715a/kube-rbac-proxy-thanos/0.log" Jul 10 00:07:52.084194 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.084169 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f3bf99cc-4504-49db-ab8b-30ced52b715a/init-config-reloader/0.log" Jul 10 00:07:52.110735 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.110708 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-x6w6j_2b4c2f26-5ff9-49b3-91fa-729707e7a2eb/prometheus-operator/0.log" Jul 10 00:07:52.132135 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.132108 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-x6w6j_2b4c2f26-5ff9-49b3-91fa-729707e7a2eb/kube-rbac-proxy/0.log" Jul 10 00:07:52.155512 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.155485 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-f7rdg_45b9e056-37fd-4765-bd6d-ea3e99d04da8/prometheus-operator-admission-webhook/0.log" Jul 10 00:07:52.182822 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.182795 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc7c784b8-9slq8_88e3646d-e876-4e03-94a8-2917f3fbeec8/telemeter-client/0.log" Jul 10 00:07:52.207966 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.207928 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc7c784b8-9slq8_88e3646d-e876-4e03-94a8-2917f3fbeec8/reload/0.log" Jul 10 00:07:52.231389 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.231326 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc7c784b8-9slq8_88e3646d-e876-4e03-94a8-2917f3fbeec8/kube-rbac-proxy/0.log" Jul 10 00:07:52.763236 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.763190 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 10 00:07:52.771232 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:52.771184 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jdkzc_eb3d9b13-8e9b-4501-883a-a8742ae6068a/ovn-acl-logging/0.log" Jul 10 00:07:54.031708 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.031657 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b"] Jul 10 00:07:54.032092 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.032070 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45096988-8878-4c1c-82b4-09af56884749" containerName="test-curl-7224-140222186908528" Jul 10 00:07:54.032092 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.032082 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="45096988-8878-4c1c-82b4-09af56884749" containerName="test-curl-7224-140222186908528" Jul 10 00:07:54.032167 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.032140 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="45096988-8878-4c1c-82b4-09af56884749" containerName="test-curl-7224-140222186908528" Jul 10 00:07:54.036654 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.036634 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.039863 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.039839 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9scwp\"/\"openshift-service-ca.crt\"" Jul 10 00:07:54.039999 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.039922 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9scwp\"/\"default-dockercfg-vnv75\"" Jul 10 00:07:54.040073 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.039857 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9scwp\"/\"kube-root-ca.crt\"" Jul 10 00:07:54.044550 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.044526 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b"] Jul 10 00:07:54.148380 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.148346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-sys\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.148551 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.148453 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-proc\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.148551 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.148485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-podres\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.148551 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.148525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-lib-modules\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.148681 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.148555 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-kube-api-access-g2r25\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249305 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-sys\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249503 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-proc\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249503 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-podres\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249503 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249410 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-lib-modules\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249503 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-sys\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249503 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249435 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-kube-api-access-g2r25\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249503 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-proc\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249720 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249539 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-podres\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.249720 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.249544 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-lib-modules\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.256979 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.256956 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2r25\" (UniqueName: \"kubernetes.io/projected/9fe51a7e-2598-4dab-9ecf-2cc728e0a47d-kube-api-access-g2r25\") pod \"perf-node-gather-daemonset-hdf4b\" (UID: \"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d\") " pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.348692 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.348602 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.480354 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.480252 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b"] Jul 10 00:07:54.482702 ip-10-0-143-88 kubenswrapper[2562]: W0710 00:07:54.482673 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9fe51a7e_2598_4dab_9ecf_2cc728e0a47d.slice/crio-c3291a4761b5854179ebc7057ea8cf0872197c72d0027041f965cc35e060c67d WatchSource:0}: Error finding container c3291a4761b5854179ebc7057ea8cf0872197c72d0027041f965cc35e060c67d: Status 404 returned error can't find the container with id c3291a4761b5854179ebc7057ea8cf0872197c72d0027041f965cc35e060c67d Jul 10 00:07:54.679186 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.679089 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" event={"ID":"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d","Type":"ContainerStarted","Data":"6845371010ddcde074b50671218e9cb86665e45f0c31dbc19cb0c4d72cd57208"} Jul 10 00:07:54.679186 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.679141 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" event={"ID":"9fe51a7e-2598-4dab-9ecf-2cc728e0a47d","Type":"ContainerStarted","Data":"c3291a4761b5854179ebc7057ea8cf0872197c72d0027041f965cc35e060c67d"} Jul 10 00:07:54.679186 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.679171 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:07:54.696939 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.696889 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" podStartSLOduration=0.696872805 podStartE2EDuration="696.872805ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:07:54.694663572 +0000 UTC m=+2402.575415533" watchObservedRunningTime="2026-07-10 00:07:54.696872805 +0000 UTC m=+2402.577624826" Jul 10 00:07:54.771731 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:54.771703 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75965c4cc9-lnxkv_fa222157-09d7-4aa8-9577-5e34f775c20e/console/0.log" Jul 10 00:07:56.371460 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:56.371432 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nsrv8_a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c/dns/0.log" Jul 10 00:07:56.400318 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:56.400292 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nsrv8_a7d4fc34-87c4-479c-9b9e-dc7e6d4b4b1c/kube-rbac-proxy/0.log" Jul 10 00:07:56.577521 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:56.577492 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-t59c6_8d45b59a-6c66-4708-afb4-0efcbf6ca153/dns-node-resolver/0.log" Jul 10 00:07:57.089166 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:57.089136 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-547f948bc6-nmfj5_cb779325-7f6b-4c2b-bb97-5db2d4883954/registry/0.log" Jul 10 00:07:57.150830 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:57.150795 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g8gb8_6cdbbd61-cd46-4dea-a427-322f8815faca/node-ca/0.log" Jul 10 00:07:58.184938 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:58.184911 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-96c2g_fc7a0613-0f47-48e0-8ccf-608fc7c0f535/kube-auth-proxy/0.log" Jul 10 00:07:58.977365 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:58.977336 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7zb56_8ab4bf87-1ebf-4b1f-b483-65ecd08136fb/serve-healthcheck-canary/0.log" Jul 10 00:07:59.553272 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:59.553248 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwjxn_0a383ff5-7316-487d-80c8-93f4ca9c6fde/kube-rbac-proxy/0.log" Jul 10 00:07:59.571833 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:59.571812 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwjxn_0a383ff5-7316-487d-80c8-93f4ca9c6fde/exporter/0.log" Jul 10 00:07:59.593062 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:07:59.593022 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwjxn_0a383ff5-7316-487d-80c8-93f4ca9c6fde/extractor/0.log" Jul 10 00:08:00.693814 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:00.693785 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9scwp/perf-node-gather-daemonset-hdf4b" Jul 10 00:08:02.285711 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:02.285680 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-qbnb6_7cfa21a5-a002-4495-9d8c-189edb2e0847/manager/0.log" Jul 10 00:08:02.302719 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:02.302695 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-c55df6879-qbnb6_7cfa21a5-a002-4495-9d8c-189edb2e0847/copy-manifests/0.log" Jul 10 00:08:03.714619 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:03.714552 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7c889b46bf-x6kbw_2acbbd40-8570-436c-9ec9-8e21403d663a/manager/0.log" Jul 10 00:08:05.311924 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:05.311836 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-96c2g_fc7a0613-0f47-48e0-8ccf-608fc7c0f535/kube-auth-proxy/0.log" Jul 10 00:08:06.820745 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:06.820719 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6dff7df579-96c2g_fc7a0613-0f47-48e0-8ccf-608fc7c0f535/kube-auth-proxy/0.log" Jul 10 00:08:08.872851 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:08.872814 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-597qf_d6b2446b-4b1a-4954-97c7-531a809ec111/migrator/0.log" Jul 10 00:08:08.891129 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:08.891101 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-597qf_d6b2446b-4b1a-4954-97c7-531a809ec111/graceful-termination/0.log" Jul 10 00:08:10.174263 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.174231 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xx52_ef2cbebe-d114-4c91-b31f-9d2698990faa/kube-multus-additional-cni-plugins/0.log" Jul 10 00:08:10.194145 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.194123 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xx52_ef2cbebe-d114-4c91-b31f-9d2698990faa/egress-router-binary-copy/0.log" Jul 10 00:08:10.212629 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.212608 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xx52_ef2cbebe-d114-4c91-b31f-9d2698990faa/cni-plugins/0.log" Jul 10 00:08:10.231298 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.231276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xx52_ef2cbebe-d114-4c91-b31f-9d2698990faa/bond-cni-plugin/0.log" Jul 10 00:08:10.251997 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.251974 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xx52_ef2cbebe-d114-4c91-b31f-9d2698990faa/routeoverride-cni/0.log" Jul 10 00:08:10.272744 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.272718 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xx52_ef2cbebe-d114-4c91-b31f-9d2698990faa/whereabouts-cni-bincopy/0.log" Jul 10 00:08:10.292169 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.292142 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2xx52_ef2cbebe-d114-4c91-b31f-9d2698990faa/whereabouts-cni/0.log" Jul 10 00:08:10.629602 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.629574 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hvswt_c3f7c514-15d5-4ff6-a32f-de5e43a59120/kube-multus/0.log" Jul 10 00:08:10.647631 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.647605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fz2qx_d7f4db34-e093-4e81-9bde-54b01feb897f/network-metrics-daemon/0.log" Jul 10 00:08:10.664594 ip-10-0-143-88 kubenswrapper[2562]: I0710 00:08:10.664560 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fz2qx_d7f4db34-e093-4e81-9bde-54b01feb897f/kube-rbac-proxy/0.log"