Jul 21 16:59:57.793824 ip-10-0-134-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 16:59:57.793835 ip-10-0-134-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 16:59:57.793842 ip-10-0-134-97 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 16:59:57.794076 ip-10-0-134-97 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 17:00:07.937596 ip-10-0-134-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 17:00:07.937615 ip-10-0-134-97 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 9a1c7773ad66460696fa247701f5b4d6 -- Jul 21 17:02:10.191856 ip-10-0-134-97 systemd[1]: Starting Kubernetes Kubelet... Jul 21 17:02:10.625520 ip-10-0-134-97 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 17:02:10.625520 ip-10-0-134-97 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 17:02:10.625520 ip-10-0-134-97 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 17:02:10.625520 ip-10-0-134-97 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 17:02:10.625520 ip-10-0-134-97 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 17:02:10.627038 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.626930 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 17:02:10.632560 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632533 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 17:02:10.632560 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632560 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632563 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632567 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632570 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632576 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632580 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632583 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632587 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632589 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632592 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632595 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632598 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632606 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632609 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632611 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632614 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632616 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632619 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632621 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 17:02:10.632622 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632624 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632627 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632630 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632632 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632636 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632638 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632641 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632643 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632646 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632648 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632651 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632654 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632656 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632660 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632663 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632666 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632669 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632671 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632674 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632676 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 17:02:10.633176 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632679 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632682 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632684 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632686 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632690 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632694 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632697 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632700 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632703 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632705 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632708 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632711 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632713 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632716 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632720 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632723 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632726 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632728 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632731 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 17:02:10.633712 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632733 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632736 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632740 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632742 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632745 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632747 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632750 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632752 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632755 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632757 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632760 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632762 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632765 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632767 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632770 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632773 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632775 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632778 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632780 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632783 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 17:02:10.634211 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632785 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632788 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632790 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632792 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632795 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632797 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632800 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632803 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632806 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632808 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632811 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632813 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632816 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632818 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632821 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632823 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632826 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632829 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632831 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 17:02:10.634736 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.632834 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 17:02:10.635532 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635516 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 17:02:10.635532 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635531 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 17:02:10.635532 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635549 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635554 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635558 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635561 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635563 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635566 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635570 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635573 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635575 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635578 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635581 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635584 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635586 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635590 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635592 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635595 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635597 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635600 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635603 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 17:02:10.635636 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635605 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635608 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635610 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635623 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635627 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635630 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635633 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635636 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635638 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635641 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635644 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635647 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635649 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635652 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635655 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635657 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635660 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635662 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635665 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 17:02:10.636093 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635667 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635670 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635672 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635675 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635681 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635684 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635687 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635690 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635695 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635698 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635702 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635705 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635707 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635711 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635715 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635718 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635721 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635724 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635726 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 17:02:10.636576 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635729 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635732 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635734 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635737 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635739 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635742 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635744 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635747 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635749 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635752 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635755 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635757 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635760 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635762 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635765 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635767 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635770 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635773 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635775 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635778 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635780 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 17:02:10.637053 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635782 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635785 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635787 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635791 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635794 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635796 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635799 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635802 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635805 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635807 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635810 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635812 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635815 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635817 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635820 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635822 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635825 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635827 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.635830 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636200 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 17:02:10.637608 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636206 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636209 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636212 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636214 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636218 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636220 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636223 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636226 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636228 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636231 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636233 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636236 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636238 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636241 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636243 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636248 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636251 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636254 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636257 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 17:02:10.638078 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636261 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636264 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636266 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636269 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636271 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636274 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636276 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636279 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636281 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636284 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636286 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636289 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636291 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636293 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636296 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636298 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636301 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636303 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636307 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 17:02:10.638555 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636310 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636313 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636316 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636319 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636322 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636325 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636328 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636330 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636333 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636335 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636342 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636345 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636347 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636350 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636352 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636355 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636358 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636360 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636364 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636367 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 17:02:10.639049 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636370 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636374 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636376 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636380 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636382 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636385 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636388 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636390 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636393 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636397 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636399 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636402 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636404 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636407 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636409 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636412 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636414 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636417 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636419 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636422 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 17:02:10.639515 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636425 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636427 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636430 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636432 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636435 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636438 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636440 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636443 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636445 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636448 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636451 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636453 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636456 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636458 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636461 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636464 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636466 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636468 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636471 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.636473 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 17:02:10.639997 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636564 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636573 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636580 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636585 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636589 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636592 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636596 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636600 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636603 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636606 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636610 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636613 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636616 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636619 2562 flags.go:64] FLAG: --cgroup-root="" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636621 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636624 2562 flags.go:64] FLAG: --client-ca-file="" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636628 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636631 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636641 2562 flags.go:64] FLAG: --cluster-domain="" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636644 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636647 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636650 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636653 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636660 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 17:02:10.640489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636663 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636667 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636670 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636673 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636676 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636679 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636682 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636685 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636689 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636692 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636695 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636698 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636701 2562 flags.go:64] FLAG: --enable-server="true" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636703 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636708 2562 flags.go:64] FLAG: --event-burst="100" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636711 2562 flags.go:64] FLAG: --event-qps="50" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636714 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636717 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636720 2562 flags.go:64] FLAG: --eviction-hard="" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636723 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636726 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636728 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636731 2562 flags.go:64] FLAG: --eviction-soft="" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636734 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636737 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 17:02:10.641068 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636744 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636747 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636750 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636753 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636755 2562 flags.go:64] FLAG: --feature-gates="" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636759 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636762 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636765 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636768 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636771 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636773 2562 flags.go:64] FLAG: --help="false" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636776 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-134-97.ec2.internal" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636779 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636782 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636785 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636788 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636792 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636795 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636797 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636800 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636803 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636806 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636809 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636812 2562 flags.go:64] FLAG: --kube-reserved="" Jul 21 17:02:10.641701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636815 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636817 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636820 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636823 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636826 2562 flags.go:64] FLAG: --lock-file="" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636829 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636831 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636834 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636841 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636844 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636847 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636850 2562 flags.go:64] FLAG: --logging-format="text" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636853 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636856 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636859 2562 flags.go:64] FLAG: --manifest-url="" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636861 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636866 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636869 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636872 2562 flags.go:64] FLAG: --max-pods="110" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636875 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636878 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636881 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636884 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636887 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636890 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 17:02:10.642278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636893 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636900 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636902 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636905 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636908 2562 flags.go:64] FLAG: --pod-cidr="" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636911 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636916 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636919 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636922 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636925 2562 flags.go:64] FLAG: --port="10250" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636928 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636931 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e56b0d91288ea651" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636934 2562 flags.go:64] FLAG: --qos-reserved="" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636937 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636940 2562 flags.go:64] FLAG: --register-node="true" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636942 2562 flags.go:64] FLAG: --register-with-taints="" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636947 2562 flags.go:64] FLAG: --registry-burst="10" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636951 2562 flags.go:64] FLAG: --registry-qps="5" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636954 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636956 2562 flags.go:64] FLAG: --reserved-memory="" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636960 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636963 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636966 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636969 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636972 2562 flags.go:64] FLAG: --runonce="false" Jul 21 17:02:10.642909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636975 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636978 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636981 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636983 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636986 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636989 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636992 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636995 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.636998 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637000 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637003 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637006 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637009 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637015 2562 flags.go:64] FLAG: --system-cgroups="" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637017 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637022 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637025 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637027 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637031 2562 flags.go:64] FLAG: --tls-min-version="" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637034 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637036 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637039 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637042 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637046 2562 flags.go:64] FLAG: --v="2" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637050 2562 flags.go:64] FLAG: --version="false" Jul 21 17:02:10.643494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637054 2562 flags.go:64] FLAG: --vmodule="" Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637058 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637061 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637195 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637200 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637203 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637208 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637211 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637215 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637218 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637221 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637225 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637228 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637232 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637235 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637237 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637240 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637242 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637245 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 17:02:10.644139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637248 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637252 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637254 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637257 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637259 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637262 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637264 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637267 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637269 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637272 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637275 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637278 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637281 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637284 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637287 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637290 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637292 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637295 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637297 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637300 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 17:02:10.644692 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637303 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637305 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637308 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637311 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637313 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637315 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637318 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637320 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637323 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637325 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637328 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637330 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637333 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637336 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637339 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637341 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637344 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637346 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637349 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637351 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 17:02:10.645174 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637354 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637356 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637359 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637362 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637365 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637367 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637370 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637373 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637376 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637378 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637380 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637383 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637386 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637388 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637391 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637393 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637395 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637398 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637400 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637403 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 17:02:10.645679 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637405 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637408 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637410 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637413 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637415 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637419 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637421 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637424 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637427 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637429 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637431 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637434 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637438 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637441 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637444 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637448 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637451 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637453 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637456 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 17:02:10.646153 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637458 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637461 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637464 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.637466 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.637917 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.644267 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.644283 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.644831 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.644838 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644893 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644897 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644901 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644904 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644907 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 17:02:10.646630 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644910 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644913 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644916 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644919 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644921 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644924 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644927 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644929 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644932 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644934 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644937 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644941 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644945 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644948 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644950 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644953 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644955 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644958 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644960 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 17:02:10.646967 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644963 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644966 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644968 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644971 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644974 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644977 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644980 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644982 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644985 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644987 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644989 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644992 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644994 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644997 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.644999 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645001 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645004 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645007 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645009 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 17:02:10.647419 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645011 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645014 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645016 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645020 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645024 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645027 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645029 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645032 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645034 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645037 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645039 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645042 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645044 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645047 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645050 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645053 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645055 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645057 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645060 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645063 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 17:02:10.647895 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645065 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645068 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645070 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645073 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645075 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645078 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645080 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645083 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645085 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645087 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645090 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645092 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645096 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645098 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645100 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645104 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645107 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645111 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645114 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645117 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 17:02:10.648379 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645120 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645122 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645125 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645127 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645130 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645132 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645135 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645138 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645140 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645152 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645155 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645158 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645161 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645164 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645166 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 17:02:10.648871 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645169 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.645174 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645287 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645291 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645294 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645297 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645299 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645302 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645305 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645308 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645310 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645313 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645315 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645318 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645321 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 17:02:10.649238 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645323 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645325 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645328 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645330 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645333 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645336 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645338 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645340 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645343 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645345 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645348 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645351 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645354 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645357 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645359 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645361 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645364 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645366 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645369 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645371 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 17:02:10.649690 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645374 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645376 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645379 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645381 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645383 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645386 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645388 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645391 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645393 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645395 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645398 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645400 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645403 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645406 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645408 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645410 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645413 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645416 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645418 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645421 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 17:02:10.650154 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645423 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645427 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645430 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645433 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645435 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645438 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645441 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645444 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645447 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645449 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645452 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645454 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645457 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645459 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645461 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645464 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645467 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645469 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645473 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 17:02:10.650638 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645475 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645478 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645480 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645483 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645485 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645488 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645490 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645492 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645495 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645497 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645500 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645502 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645505 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645507 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645514 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645516 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645519 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645522 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645527 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645529 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 17:02:10.651083 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645531 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645548 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645552 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645556 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645559 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645561 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:10.645564 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.645570 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.645664 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.647506 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.648654 2562 server.go:1019] "Starting client certificate rotation" Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.648751 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.648779 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 17:02:10.651640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.651347 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 17:02:10.652482 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.652461 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 17:02:10.659000 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.658982 2562 log.go:25] "Validated CRI v1 runtime API" Jul 21 17:02:10.660830 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.660816 2562 log.go:25] "Validated CRI v1 image API" Jul 21 17:02:10.662210 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.662196 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 17:02:10.665006 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.664982 2562 fs.go:135] Filesystem UUIDs: map[0bdb903d-de33-406a-ac52-f433d1be4f25:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dd1b45e6-a012-4f0c-b1a6-0a163571d3ea:/dev/nvme0n1p3] Jul 21 17:02:10.665067 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.665005 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 17:02:10.671277 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.671165 2562 manager.go:216] Machine: {Timestamp:2026-07-21 17:02:10.670107417 +0000 UTC m=+0.274819508 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095540 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cee4aa1c920b87d0de0a3b08125a4 SystemUUID:ec2cee4a-a1c9-20b8-7d0d-e0a3b08125a4 BootID:9a1c7773-ad66-4606-96fa-247701f5b4d6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:39:94:5e:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:39:94:5e:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:c0:85:2d:e6:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 17:02:10.671277 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.671268 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 17:02:10.671437 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.671413 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 17:02:10.673975 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.673950 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 17:02:10.674101 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.673978 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 17:02:10.675035 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.675025 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 17:02:10.675072 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.675037 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 17:02:10.675072 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.675045 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 17:02:10.675594 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.675574 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 17:02:10.676690 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.676680 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 17:02:10.679633 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.679621 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 21 17:02:10.679738 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.679729 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 17:02:10.680405 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.680396 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 21 17:02:10.680434 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.680410 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 17:02:10.680434 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.680422 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 17:02:10.680434 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.680432 2562 kubelet.go:398] "Adding apiserver pod source" Jul 21 17:02:10.680571 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.680442 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 17:02:10.680758 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.680748 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 17:02:10.680790 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.680765 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 17:02:10.682612 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.682590 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 17:02:10.684179 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.684161 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 17:02:10.684475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.684463 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 17:02:10.684515 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.684493 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 17:02:10.685579 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685567 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685583 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685590 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685595 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685604 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685609 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685617 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685623 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 17:02:10.685631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685630 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 17:02:10.685836 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685635 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 17:02:10.685836 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685653 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 17:02:10.685836 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.685659 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 17:02:10.686903 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.686892 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 17:02:10.686940 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.686905 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 17:02:10.690181 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:10.690151 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 17:02:10.690181 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:10.690169 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 17:02:10.690322 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.690227 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 17:02:10.690322 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.690255 2562 server.go:1268] "Started kubelet" Jul 21 17:02:10.690401 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.690330 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 17:02:10.690503 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.690410 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 17:02:10.690598 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.690580 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 17:02:10.691504 ip-10-0-134-97 systemd[1]: Started Kubernetes Kubelet. Jul 21 17:02:10.692174 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.692144 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 17:02:10.692744 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.692728 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 21 17:02:10.695663 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:10.694865 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-97.ec2.internal.18c45cbfaf026455 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-97.ec2.internal,UID:ip-10-0-134-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-97.ec2.internal,},FirstTimestamp:2026-07-21 17:02:10.690237525 +0000 UTC m=+0.294949598,LastTimestamp:2026-07-21 17:02:10.690237525 +0000 UTC m=+0.294949598,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-97.ec2.internal,}" Jul 21 17:02:10.695903 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:10.695887 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 17:02:10.697759 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.697743 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 17:02:10.697859 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:10.697835 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:10.697944 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.697907 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 17:02:10.698359 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698344 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 17:02:10.698359 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698361 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 17:02:10.698474 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698363 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 17:02:10.698639 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698626 2562 factory.go:55] Registering systemd factory Jul 21 17:02:10.698639 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698640 2562 factory.go:223] Registration of the systemd container factory successfully Jul 21 17:02:10.698835 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698820 2562 factory.go:153] Registering CRI-O factory Jul 21 17:02:10.698883 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698838 2562 factory.go:223] Registration of the crio container factory successfully Jul 21 17:02:10.698931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698885 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 17:02:10.698931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698908 2562 factory.go:103] Registering Raw factory Jul 21 17:02:10.698931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.698918 2562 manager.go:1196] Started watching for new ooms in manager Jul 21 17:02:10.699380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.699369 2562 manager.go:349] Starting recovery of all containers Jul 21 17:02:10.699450 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.699389 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 21 17:02:10.699450 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.699400 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 21 17:02:10.700950 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:10.700907 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 17:02:10.701072 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:10.701038 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 17:02:10.712994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.712905 2562 manager.go:354] Recovery completed Jul 21 17:02:10.715936 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.715912 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kdk59" Jul 21 17:02:10.717377 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.717366 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 17:02:10.719529 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.719513 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 17:02:10.719529 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.719555 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 17:02:10.719529 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.719569 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 17:02:10.720119 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.720105 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 17:02:10.720119 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.720118 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 17:02:10.720209 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.720137 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 21 17:02:10.720209 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.720119 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 17:02:10.720858 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.720842 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 17:02:10.720908 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.720868 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 17:02:10.720908 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.720878 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 17:02:10.721362 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.721342 2562 policy_none.go:49] "None policy: Start" Jul 21 17:02:10.721424 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.721369 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 17:02:10.721424 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.721383 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 17:02:10.722522 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.722508 2562 policy_none.go:47] "Start" Jul 21 17:02:10.724953 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:10.724936 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kdk59" Jul 21 17:02:12.245803 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.245675 2562 manager.go:337] "Starting Device Plugin manager" Jul 21 17:02:12.245901 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.245820 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 17:02:12.245901 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.245830 2562 server.go:87] "Starting device plugin registration server" Jul 21 17:02:12.246116 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.246092 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 17:02:12.246168 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.246119 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 17:02:12.246215 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.246178 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 17:02:12.246334 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.246319 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 17:02:12.246334 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.246334 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 17:02:12.246863 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.246839 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 17:02:12.246953 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.246890 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.346574 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.346551 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 17:02:12.347968 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.347507 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 17:02:12.347968 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.347535 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 17:02:12.347968 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.347546 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 17:02:12.347968 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.347578 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.361505 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.361486 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.361554 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.361512 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-97.ec2.internal\": node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.370087 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.370025 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 17:02:12.371345 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.371330 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 17:02:12.371437 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.371372 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 17:02:12.371437 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.371401 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 17:02:12.371514 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.371440 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 17:02:12.373748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.373727 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 17:02:12.377317 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.377302 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.472159 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.472133 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal"] Jul 21 17:02:12.472244 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.472193 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 17:02:12.473074 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.473043 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 17:02:12.473074 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.473069 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 17:02:12.473074 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.473079 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 17:02:12.474621 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.474609 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 17:02:12.474795 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.474782 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.474835 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.474815 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 17:02:12.475364 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.475339 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 17:02:12.475416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.475366 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 17:02:12.475416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.475380 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 17:02:12.475416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.475409 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 17:02:12.475512 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.475430 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 17:02:12.475512 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.475445 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 17:02:12.476534 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.476521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.476581 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.476546 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 17:02:12.477160 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.477144 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 17:02:12.477243 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.477169 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 17:02:12.477243 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.477181 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 17:02:12.477368 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.477354 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.491672 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.491650 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d8c58923675c06522e424156ee02f34-config\") pod \"kube-apiserver-proxy-ip-10-0-134-97.ec2.internal\" (UID: \"4d8c58923675c06522e424156ee02f34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.491768 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.491684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/899bf032b8c68fb92d0272de53c47dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal\" (UID: \"899bf032b8c68fb92d0272de53c47dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.491768 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.491718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/899bf032b8c68fb92d0272de53c47dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal\" (UID: \"899bf032b8c68fb92d0272de53c47dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.502645 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.502627 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-97.ec2.internal\" not found" node="ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.506862 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.506844 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-97.ec2.internal\" not found" node="ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.577805 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.577787 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.592485 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.592469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d8c58923675c06522e424156ee02f34-config\") pod \"kube-apiserver-proxy-ip-10-0-134-97.ec2.internal\" (UID: \"4d8c58923675c06522e424156ee02f34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.592532 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.592490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/899bf032b8c68fb92d0272de53c47dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal\" (UID: \"899bf032b8c68fb92d0272de53c47dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.592532 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.592513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/899bf032b8c68fb92d0272de53c47dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal\" (UID: \"899bf032b8c68fb92d0272de53c47dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.592607 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.592557 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4d8c58923675c06522e424156ee02f34-config\") pod \"kube-apiserver-proxy-ip-10-0-134-97.ec2.internal\" (UID: \"4d8c58923675c06522e424156ee02f34\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.592607 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.592554 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/899bf032b8c68fb92d0272de53c47dc5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal\" (UID: \"899bf032b8c68fb92d0272de53c47dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.592607 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.592592 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/899bf032b8c68fb92d0272de53c47dc5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal\" (UID: \"899bf032b8c68fb92d0272de53c47dc5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.678556 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.678503 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.778979 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.778957 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.806424 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.806407 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.809504 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:12.809489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" Jul 21 17:02:12.879724 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.879694 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:12.980282 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:12.980235 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:13.080711 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:13.080685 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:13.139233 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.139208 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 17:02:13.139586 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.139343 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 17:02:13.181645 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:13.181624 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:13.219139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.219098 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 16:57:12 +0000 UTC" deadline="2027-12-21 08:13:56.119679901 +0000 UTC" Jul 21 17:02:13.219139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.219132 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12423h11m42.900550509s" Jul 21 17:02:13.282450 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:13.282404 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-97.ec2.internal\" not found" Jul 21 17:02:13.295062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.295048 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 17:02:13.389571 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.389549 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" Jul 21 17:02:13.401224 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.401209 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 17:02:13.402698 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.402687 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" Jul 21 17:02:13.417950 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.417932 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 17:02:13.428741 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.428726 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 17:02:13.616415 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:13.616388 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899bf032b8c68fb92d0272de53c47dc5.slice/crio-a06f5d63a2105d318bfee9944ac6d894dcf1a264d6e051a8de331fbcc3f86e6f WatchSource:0}: Error finding container a06f5d63a2105d318bfee9944ac6d894dcf1a264d6e051a8de331fbcc3f86e6f: Status 404 returned error can't find the container with id a06f5d63a2105d318bfee9944ac6d894dcf1a264d6e051a8de331fbcc3f86e6f Jul 21 17:02:13.621347 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.621332 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 17:02:13.694934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:13.694912 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 17:02:13.815283 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:13.815260 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8c58923675c06522e424156ee02f34.slice/crio-2550eb29314b9ef57ab5fed2952d68b1d0831f2b07cac056c6c4015ef68df69a WatchSource:0}: Error finding container 2550eb29314b9ef57ab5fed2952d68b1d0831f2b07cac056c6c4015ef68df69a: Status 404 returned error can't find the container with id 2550eb29314b9ef57ab5fed2952d68b1d0831f2b07cac056c6c4015ef68df69a Jul 21 17:02:14.173010 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.172989 2562 apiserver.go:52] "Watching apiserver" Jul 21 17:02:14.181708 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.181688 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 17:02:14.182061 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.182043 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-mq5p8","openshift-network-operator/iptables-alerter-68crx","openshift-ovn-kubernetes/ovnkube-node-jztcs","kube-system/global-pull-secret-syncer-6xnc7","kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96","openshift-cluster-node-tuning-operator/tuned-h6k29","openshift-dns/node-resolver-clg67","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal","openshift-multus/multus-rqjbb","openshift-multus/network-metrics-daemon-9tfrw","kube-system/konnectivity-agent-qwbq2","openshift-image-registry/node-ca-5vkhj","openshift-multus/multus-additional-cni-plugins-bxpxv"] Jul 21 17:02:14.184116 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.184087 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.185705 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.185690 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:14.185781 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.185745 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:14.186444 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.186423 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 17:02:14.186444 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.186426 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 17:02:14.186602 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.186450 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xjmn8\"" Jul 21 17:02:14.187048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.187032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.188033 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.188017 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.188167 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.188087 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:14.188966 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.188953 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 17:02:14.189638 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.189491 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 17:02:14.189638 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.189512 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 17:02:14.189638 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.189496 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 17:02:14.189638 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.189594 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.190289 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.190267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 17:02:14.190846 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.190814 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 17:02:14.191751 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.191181 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 17:02:14.191751 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.191252 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hpsbv\"" Jul 21 17:02:14.193475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.192276 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 17:02:14.193475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.192364 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 17:02:14.193475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.192682 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mdvht\"" Jul 21 17:02:14.193475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.192959 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 17:02:14.193475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.193204 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 17:02:14.193475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.193414 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 17:02:14.193848 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.193613 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.194654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.194637 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.195882 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.195862 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 17:02:14.195977 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.195934 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 17:02:14.195977 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.195964 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 17:02:14.196181 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.196166 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.196297 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.196167 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4f6p8\"" Jul 21 17:02:14.196621 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.196604 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zgll9\"" Jul 21 17:02:14.196688 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.196609 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 17:02:14.196739 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.196626 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 17:02:14.197658 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.197638 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.198114 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.198086 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 17:02:14.198303 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.198291 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ll7nw\"" Jul 21 17:02:14.198364 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.198334 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 17:02:14.198457 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.198444 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 17:02:14.198938 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.198922 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.199472 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.199455 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 17:02:14.199967 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.199947 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 17:02:14.200050 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200027 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mn262\"" Jul 21 17:02:14.200133 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200059 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 17:02:14.200328 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200313 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:14.200407 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.200384 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:14.200766 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysctl-conf\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.200833 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-systemd\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.200891 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j28pt\" (UniqueName: \"kubernetes.io/projected/c326346e-e5d3-4618-ab4c-9b27c8946dc2-kube-api-access-j28pt\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:14.200891 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-kubelet\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.200993 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-run-ovn-kubernetes\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.200993 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-system-cni-dir\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.200993 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200966 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-kubernetes\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.200993 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200981 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 17:02:14.200993 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.200983 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55d47003-e88d-472b-b137-e7cc993956ac-tmp\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201184 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201000 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qlmtx\"" Jul 21 17:02:14.201184 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-ovnkube-config\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201184 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.201184 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201184 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysctl-d\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201184 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201150 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-sys\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201184 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201169 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-systemd-units\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201221 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-slash\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201262 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-systemd\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201307 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-modprobe-d\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201320 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysconfig\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55d47003-e88d-472b-b137-e7cc993956ac-etc-tuned\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201374 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-run-netns\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-var-lib-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-etc-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-cni-bin\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201520 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27bx\" (UniqueName: \"kubernetes.io/projected/55d47003-e88d-472b-b137-e7cc993956ac-kube-api-access-r27bx\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201538 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-ovnkube-script-lib\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4835f1ed-ad53-45e6-b120-b08aa18e8baf-kubelet-config\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201628 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4835f1ed-ad53-45e6-b120-b08aa18e8baf-dbus\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201673 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-os-release\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201726 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-var-lib-kubelet\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-ovn\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201762 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-env-overrides\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.201784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc445252-4547-4ccf-961f-4f9d50accba2-ovn-node-metrics-cert\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfq97\" (UniqueName: \"kubernetes.io/projected/fc445252-4547-4ccf-961f-4f9d50accba2-kube-api-access-cfq97\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201844 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cnibin\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201873 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dh4\" (UniqueName: \"kubernetes.io/projected/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-kube-api-access-p8dh4\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201897 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-run\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-lib-modules\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201942 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-node-log\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201969 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-log-socket\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.201997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-cni-netd\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.202042 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.202069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-host\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.202129 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.202293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.202164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.203067 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.203049 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 17:02:14.203541 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.203528 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 17:02:14.203613 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.203590 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 17:02:14.203674 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.203655 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5r9rg\"" Jul 21 17:02:14.232266 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.232247 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wsb6m" Jul 21 17:02:14.242934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.242920 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wsb6m" Jul 21 17:02:14.290002 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.289976 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 17:02:14.302846 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302704 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b668f-679a-4dc5-9fae-2e3cf8c109d4-konnectivity-ca\") pod \"konnectivity-agent-qwbq2\" (UID: \"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4\") " pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.302846 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302743 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-ovnkube-script-lib\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.302846 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-host-slash\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.302846 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-multus-certs\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.302846 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc445252-4547-4ccf-961f-4f9d50accba2-ovn-node-metrics-cert\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cnibin\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dh4\" (UniqueName: \"kubernetes.io/projected/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-kube-api-access-p8dh4\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-conf-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-run\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-node-log\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-cnibin\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.302993 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-etc-kubernetes\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-device-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrtx\" (UniqueName: \"kubernetes.io/projected/41c19cc9-d39e-424f-9c99-28f3d81a8522-kube-api-access-2nrtx\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303095 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-iptables-alerter-script\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysctl-conf\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.303176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303164 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-systemd\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-kubelet\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-system-cni-dir\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303245 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-registration-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-hostroot\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55d47003-e88d-472b-b137-e7cc993956ac-tmp\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-ovnkube-config\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303372 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cnibin\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-system-cni-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303409 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-ovnkube-script-lib\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303427 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-cni-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303426 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysctl-d\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303496 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-systemd-units\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303505 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysctl-conf\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.303994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-slash\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-run\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-systemd\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-var-lib-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303623 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-systemd\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303623 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-systemd-units\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-systemd\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303657 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-slash\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfq97\" (UniqueName: \"kubernetes.io/projected/fc445252-4547-4ccf-961f-4f9d50accba2-kube-api-access-cfq97\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-kubelet\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303735 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-system-cni-dir\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303743 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-modprobe-d\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303774 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysconfig\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-var-lib-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303767 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-node-log\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55d47003-e88d-472b-b137-e7cc993956ac-etc-tuned\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-run-netns\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysctl-d\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.304801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303841 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-sysconfig\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303885 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-modprobe-d\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-run-netns\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-etc-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.303973 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-cni-bin\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304010 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-cni-bin\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304063 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-etc-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4835f1ed-ad53-45e6-b120-b08aa18e8baf-dbus\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304159 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-ovnkube-config\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27bx\" (UniqueName: \"kubernetes.io/projected/55d47003-e88d-472b-b137-e7cc993956ac-kube-api-access-r27bx\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304220 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4835f1ed-ad53-45e6-b120-b08aa18e8baf-kubelet-config\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304249 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-os-release\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304316 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-os-release\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4835f1ed-ad53-45e6-b120-b08aa18e8baf-dbus\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.305870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304436 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4835f1ed-ad53-45e6-b120-b08aa18e8baf-kubelet-config\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304475 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-sys-fs\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304540 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwmj2\" (UniqueName: \"kubernetes.io/projected/66b23d89-c2ad-4532-a99e-ad1eee9e3905-kube-api-access-qwmj2\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304580 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c19cc9-d39e-424f-9c99-28f3d81a8522-tmp-dir\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304614 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-var-lib-kubelet\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-ovn\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304671 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtgwm\" (UniqueName: \"kubernetes.io/projected/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-kube-api-access-rtgwm\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304705 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-ovn\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cfa7c62-ef41-4665-9794-751565d61390-host\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-cni-bin\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304753 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-var-lib-kubelet\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304779 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pc5\" (UniqueName: \"kubernetes.io/projected/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-kube-api-access-g6pc5\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a6b668f-679a-4dc5-9fae-2e3cf8c109d4-agent-certs\") pod \"konnectivity-agent-qwbq2\" (UID: \"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4\") " pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-lib-modules\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.306666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-log-socket\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-cni-netd\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304898 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304921 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-socket-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-cni-netd\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41c19cc9-d39e-424f-9c99-28f3d81a8522-hosts-file\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-daemon-config\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304994 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-host\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305007 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-lib-modules\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.305014 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.304921 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-log-socket\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-host\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.305079 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret podName:4835f1ed-ad53-45e6-b120-b08aa18e8baf nodeName:}" failed. No retries permitted until 2026-07-21 17:02:14.805059723 +0000 UTC m=+2.919206706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret") pod "global-pull-secret-syncer-6xnc7" (UID: "4835f1ed-ad53-45e6-b120-b08aa18e8baf") : object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cfa7c62-ef41-4665-9794-751565d61390-serviceca\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltjkb\" (UniqueName: \"kubernetes.io/projected/5cfa7c62-ef41-4665-9794-751565d61390-kube-api-access-ltjkb\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.307544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j28pt\" (UniqueName: \"kubernetes.io/projected/c326346e-e5d3-4618-ab4c-9b27c8946dc2-kube-api-access-j28pt\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-run-ovn-kubernetes\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-socket-dir-parent\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-kubernetes\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-kubelet\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-sys\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305373 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-env-overrides\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-cni-binary-copy\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305444 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-k8s-cni-cncf-io\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-os-release\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305491 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-netns\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-cni-multus\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305603 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-sys\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.308380 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.305631 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305647 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-host-run-ovn-kubernetes\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.305676 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs podName:c326346e-e5d3-4618-ab4c-9b27c8946dc2 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:14.805659918 +0000 UTC m=+2.919806894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs") pod "network-metrics-daemon-9tfrw" (UID: "c326346e-e5d3-4618-ab4c-9b27c8946dc2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55d47003-e88d-472b-b137-e7cc993956ac-etc-kubernetes\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305759 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fc445252-4547-4ccf-961f-4f9d50accba2-run-openvswitch\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305902 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.305986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fc445252-4547-4ccf-961f-4f9d50accba2-env-overrides\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.307553 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/55d47003-e88d-472b-b137-e7cc993956ac-etc-tuned\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.307569 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55d47003-e88d-472b-b137-e7cc993956ac-tmp\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.309080 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.307834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fc445252-4547-4ccf-961f-4f9d50accba2-ovn-node-metrics-cert\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.320953 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.320875 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfq97\" (UniqueName: \"kubernetes.io/projected/fc445252-4547-4ccf-961f-4f9d50accba2-kube-api-access-cfq97\") pod \"ovnkube-node-jztcs\" (UID: \"fc445252-4547-4ccf-961f-4f9d50accba2\") " pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.323307 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.323263 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27bx\" (UniqueName: \"kubernetes.io/projected/55d47003-e88d-472b-b137-e7cc993956ac-kube-api-access-r27bx\") pod \"tuned-h6k29\" (UID: \"55d47003-e88d-472b-b137-e7cc993956ac\") " pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.323693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.323671 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dh4\" (UniqueName: \"kubernetes.io/projected/dfb9878d-5a06-4969-82ea-8bb9aee29f7c-kube-api-access-p8dh4\") pod \"multus-additional-cni-plugins-bxpxv\" (UID: \"dfb9878d-5a06-4969-82ea-8bb9aee29f7c\") " pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.325680 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.325649 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j28pt\" (UniqueName: \"kubernetes.io/projected/c326346e-e5d3-4618-ab4c-9b27c8946dc2-kube-api-access-j28pt\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:14.377854 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.377802 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" event={"ID":"4d8c58923675c06522e424156ee02f34","Type":"ContainerStarted","Data":"2550eb29314b9ef57ab5fed2952d68b1d0831f2b07cac056c6c4015ef68df69a"} Jul 21 17:02:14.379769 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.379740 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" event={"ID":"899bf032b8c68fb92d0272de53c47dc5","Type":"ContainerStarted","Data":"a06f5d63a2105d318bfee9944ac6d894dcf1a264d6e051a8de331fbcc3f86e6f"} Jul 21 17:02:14.406613 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406585 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrtx\" (UniqueName: \"kubernetes.io/projected/41c19cc9-d39e-424f-9c99-28f3d81a8522-kube-api-access-2nrtx\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.406725 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-iptables-alerter-script\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-registration-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406861 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-hostroot\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406915 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-system-cni-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-registration-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-cni-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406990 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-cni-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.406989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-sys-fs\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407030 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwmj2\" (UniqueName: \"kubernetes.io/projected/66b23d89-c2ad-4532-a99e-ad1eee9e3905-kube-api-access-qwmj2\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c19cc9-d39e-424f-9c99-28f3d81a8522-tmp-dir\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtgwm\" (UniqueName: \"kubernetes.io/projected/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-kube-api-access-rtgwm\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407138 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cfa7c62-ef41-4665-9794-751565d61390-host\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-etc-selinux\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-cni-bin\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.409311 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-hostroot\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pc5\" (UniqueName: \"kubernetes.io/projected/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-kube-api-access-g6pc5\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a6b668f-679a-4dc5-9fae-2e3cf8c109d4-agent-certs\") pod \"konnectivity-agent-qwbq2\" (UID: \"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4\") " pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-socket-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41c19cc9-d39e-424f-9c99-28f3d81a8522-hosts-file\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-daemon-config\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407325 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-iptables-alerter-script\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cfa7c62-ef41-4665-9794-751565d61390-serviceca\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltjkb\" (UniqueName: \"kubernetes.io/projected/5cfa7c62-ef41-4665-9794-751565d61390-kube-api-access-ltjkb\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407396 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-socket-dir-parent\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407400 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-sys-fs\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407445 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-kubelet\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407452 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-system-cni-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41c19cc9-d39e-424f-9c99-28f3d81a8522-tmp-dir\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-cni-binary-copy\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407511 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-k8s-cni-cncf-io\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407534 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-os-release\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.410221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407548 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-socket-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-netns\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-cni-multus\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b668f-679a-4dc5-9fae-2e3cf8c109d4-konnectivity-ca\") pod \"konnectivity-agent-qwbq2\" (UID: \"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4\") " pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407680 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-host-slash\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-multus-certs\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-cni-bin\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407743 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-conf-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/41c19cc9-d39e-424f-9c99-28f3d81a8522-hosts-file\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-cnibin\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-k8s-cni-cncf-io\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-etc-kubernetes\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407868 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-device-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407881 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-os-release\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-cni-multus\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407941 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-host-slash\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.407979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-multus-certs\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408021 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5cfa7c62-ef41-4665-9794-751565d61390-host\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.411086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408460 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3a6b668f-679a-4dc5-9fae-2e3cf8c109d4-konnectivity-ca\") pod \"konnectivity-agent-qwbq2\" (UID: \"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4\") " pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-cni-binary-copy\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408733 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-etc-kubernetes\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408792 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66b23d89-c2ad-4532-a99e-ad1eee9e3905-device-dir\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408830 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-run-netns\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-socket-dir-parent\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.408925 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-daemon-config\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.409178 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-host-var-lib-kubelet\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.409232 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-cnibin\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.409242 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5cfa7c62-ef41-4665-9794-751565d61390-serviceca\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.409270 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-multus-conf-dir\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.411934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.410483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3a6b668f-679a-4dc5-9fae-2e3cf8c109d4-agent-certs\") pod \"konnectivity-agent-qwbq2\" (UID: \"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4\") " pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.415646 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.415624 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 17:02:14.415759 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.415652 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 17:02:14.415759 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.415666 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nhwpk for pod openshift-network-diagnostics/network-check-target-mq5p8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:14.415759 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.415749 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk podName:aee0bcc3-658e-400b-8911-829d8af34990 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:14.915732759 +0000 UTC m=+3.029879745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nhwpk" (UniqueName: "kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk") pod "network-check-target-mq5p8" (UID: "aee0bcc3-658e-400b-8911-829d8af34990") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:14.418883 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.418822 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pc5\" (UniqueName: \"kubernetes.io/projected/0d18ee3a-5b4e-4448-9c21-5c2a535dd38a-kube-api-access-g6pc5\") pod \"multus-rqjbb\" (UID: \"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a\") " pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.419274 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.419253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtgwm\" (UniqueName: \"kubernetes.io/projected/3af9ba18-eaaa-4c45-815b-3a3139cb4d3f-kube-api-access-rtgwm\") pod \"iptables-alerter-68crx\" (UID: \"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f\") " pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.419576 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.419555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwmj2\" (UniqueName: \"kubernetes.io/projected/66b23d89-c2ad-4532-a99e-ad1eee9e3905-kube-api-access-qwmj2\") pod \"aws-ebs-csi-driver-node-mkm96\" (UID: \"66b23d89-c2ad-4532-a99e-ad1eee9e3905\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.420816 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.420795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltjkb\" (UniqueName: \"kubernetes.io/projected/5cfa7c62-ef41-4665-9794-751565d61390-kube-api-access-ltjkb\") pod \"node-ca-5vkhj\" (UID: \"5cfa7c62-ef41-4665-9794-751565d61390\") " pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.421499 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.421328 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrtx\" (UniqueName: \"kubernetes.io/projected/41c19cc9-d39e-424f-9c99-28f3d81a8522-kube-api-access-2nrtx\") pod \"node-resolver-clg67\" (UID: \"41c19cc9-d39e-424f-9c99-28f3d81a8522\") " pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.495370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.495300 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h6k29" Jul 21 17:02:14.503229 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.503205 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:14.509612 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.509587 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" Jul 21 17:02:14.509702 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.509616 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d47003_e88d_472b_b137_e7cc993956ac.slice/crio-6f5989cfe60280797fa9c491b5ab9754b114599a2277c3616a9d99c26f2bbbf2 WatchSource:0}: Error finding container 6f5989cfe60280797fa9c491b5ab9754b114599a2277c3616a9d99c26f2bbbf2: Status 404 returned error can't find the container with id 6f5989cfe60280797fa9c491b5ab9754b114599a2277c3616a9d99c26f2bbbf2 Jul 21 17:02:14.512921 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.512740 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" Jul 21 17:02:14.516585 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.516555 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc445252_4547_4ccf_961f_4f9d50accba2.slice/crio-ee7a16ee4eb77ac92475f43460fecf79f975763b27dc073c8ba2c3eccad2028b WatchSource:0}: Error finding container ee7a16ee4eb77ac92475f43460fecf79f975763b27dc073c8ba2c3eccad2028b: Status 404 returned error can't find the container with id ee7a16ee4eb77ac92475f43460fecf79f975763b27dc073c8ba2c3eccad2028b Jul 21 17:02:14.518734 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.518587 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-clg67" Jul 21 17:02:14.524023 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.524004 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5vkhj" Jul 21 17:02:14.525330 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.525303 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b23d89_c2ad_4532_a99e_ad1eee9e3905.slice/crio-f67a8d2dbfc0067e38a7890d06c22f76ee44327a22dd4b8f6e4ef015809bf5e8 WatchSource:0}: Error finding container f67a8d2dbfc0067e38a7890d06c22f76ee44327a22dd4b8f6e4ef015809bf5e8: Status 404 returned error can't find the container with id f67a8d2dbfc0067e38a7890d06c22f76ee44327a22dd4b8f6e4ef015809bf5e8 Jul 21 17:02:14.529933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.529911 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-68crx" Jul 21 17:02:14.530775 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.530752 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c19cc9_d39e_424f_9c99_28f3d81a8522.slice/crio-1ea090fbb8dc5ed9dcd51a3d610a5d46b22888556a515c5f1eac1e5f1ed8b68d WatchSource:0}: Error finding container 1ea090fbb8dc5ed9dcd51a3d610a5d46b22888556a515c5f1eac1e5f1ed8b68d: Status 404 returned error can't find the container with id 1ea090fbb8dc5ed9dcd51a3d610a5d46b22888556a515c5f1eac1e5f1ed8b68d Jul 21 17:02:14.534822 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.534803 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rqjbb" Jul 21 17:02:14.535588 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.535554 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cfa7c62_ef41_4665_9794_751565d61390.slice/crio-de0088e6ae8de25ee1ca12b32ea42a112edaa7fda36559c0c0eb20bfee7f6590 WatchSource:0}: Error finding container de0088e6ae8de25ee1ca12b32ea42a112edaa7fda36559c0c0eb20bfee7f6590: Status 404 returned error can't find the container with id de0088e6ae8de25ee1ca12b32ea42a112edaa7fda36559c0c0eb20bfee7f6590 Jul 21 17:02:14.540017 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.539784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:14.545533 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.545506 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af9ba18_eaaa_4c45_815b_3a3139cb4d3f.slice/crio-afcdb593ceb7870fc371950befc9db094732d0ea5c1e62ce0b1ea6f3cb903b3c WatchSource:0}: Error finding container afcdb593ceb7870fc371950befc9db094732d0ea5c1e62ce0b1ea6f3cb903b3c: Status 404 returned error can't find the container with id afcdb593ceb7870fc371950befc9db094732d0ea5c1e62ce0b1ea6f3cb903b3c Jul 21 17:02:14.551714 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.551680 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d18ee3a_5b4e_4448_9c21_5c2a535dd38a.slice/crio-368e9336f7dfbc5700c0ec74ed8e36db9bf463afe2f86aecb566073505a98ea3 WatchSource:0}: Error finding container 368e9336f7dfbc5700c0ec74ed8e36db9bf463afe2f86aecb566073505a98ea3: Status 404 returned error can't find the container with id 368e9336f7dfbc5700c0ec74ed8e36db9bf463afe2f86aecb566073505a98ea3 Jul 21 17:02:14.557835 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:02:14.557804 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6b668f_679a_4dc5_9fae_2e3cf8c109d4.slice/crio-45d309de0283f0f9df41f68fd2037e1b744ea9051c19a73ca1d9d0b4f9c11ed9 WatchSource:0}: Error finding container 45d309de0283f0f9df41f68fd2037e1b744ea9051c19a73ca1d9d0b4f9c11ed9: Status 404 returned error can't find the container with id 45d309de0283f0f9df41f68fd2037e1b744ea9051c19a73ca1d9d0b4f9c11ed9 Jul 21 17:02:14.652868 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.652667 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 17:02:14.811937 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.811866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:14.811937 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:14.811924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:14.812167 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.811998 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:14.812167 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.812068 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:14.812167 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.812084 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret podName:4835f1ed-ad53-45e6-b120-b08aa18e8baf nodeName:}" failed. No retries permitted until 2026-07-21 17:02:15.812062383 +0000 UTC m=+3.926209362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret") pod "global-pull-secret-syncer-6xnc7" (UID: "4835f1ed-ad53-45e6-b120-b08aa18e8baf") : object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:14.812167 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:14.812136 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs podName:c326346e-e5d3-4618-ab4c-9b27c8946dc2 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:15.812117632 +0000 UTC m=+3.926264612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs") pod "network-metrics-daemon-9tfrw" (UID: "c326346e-e5d3-4618-ab4c-9b27c8946dc2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:15.014013 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.013987 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:15.014198 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.014169 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 17:02:15.014198 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.014188 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 17:02:15.014283 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.014201 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nhwpk for pod openshift-network-diagnostics/network-check-target-mq5p8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:15.014283 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.014255 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk podName:aee0bcc3-658e-400b-8911-829d8af34990 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:16.014241441 +0000 UTC m=+4.128388404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhwpk" (UniqueName: "kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk") pod "network-check-target-mq5p8" (UID: "aee0bcc3-658e-400b-8911-829d8af34990") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:15.245048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.244935 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 16:57:14 +0000 UTC" deadline="2028-02-06 22:18:46.589417677 +0000 UTC" Jul 21 17:02:15.245048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.244992 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13565h16m31.344429985s" Jul 21 17:02:15.371732 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.371691 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:15.371904 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.371708 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:15.371904 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.371817 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:15.371904 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.371861 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:15.382930 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.382898 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwbq2" event={"ID":"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4","Type":"ContainerStarted","Data":"45d309de0283f0f9df41f68fd2037e1b744ea9051c19a73ca1d9d0b4f9c11ed9"} Jul 21 17:02:15.384061 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.384035 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqjbb" event={"ID":"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a","Type":"ContainerStarted","Data":"368e9336f7dfbc5700c0ec74ed8e36db9bf463afe2f86aecb566073505a98ea3"} Jul 21 17:02:15.385154 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.385132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-68crx" event={"ID":"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f","Type":"ContainerStarted","Data":"afcdb593ceb7870fc371950befc9db094732d0ea5c1e62ce0b1ea6f3cb903b3c"} Jul 21 17:02:15.386137 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.386097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-clg67" event={"ID":"41c19cc9-d39e-424f-9c99-28f3d81a8522","Type":"ContainerStarted","Data":"1ea090fbb8dc5ed9dcd51a3d610a5d46b22888556a515c5f1eac1e5f1ed8b68d"} Jul 21 17:02:15.387149 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.387125 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"ee7a16ee4eb77ac92475f43460fecf79f975763b27dc073c8ba2c3eccad2028b"} Jul 21 17:02:15.388404 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.388381 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vkhj" event={"ID":"5cfa7c62-ef41-4665-9794-751565d61390","Type":"ContainerStarted","Data":"de0088e6ae8de25ee1ca12b32ea42a112edaa7fda36559c0c0eb20bfee7f6590"} Jul 21 17:02:15.389464 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.389445 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerStarted","Data":"051ab390697331f4a201e408e30f5f56214cbb2098293e15f77d7eda04ecff5a"} Jul 21 17:02:15.390598 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.390568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" event={"ID":"66b23d89-c2ad-4532-a99e-ad1eee9e3905","Type":"ContainerStarted","Data":"f67a8d2dbfc0067e38a7890d06c22f76ee44327a22dd4b8f6e4ef015809bf5e8"} Jul 21 17:02:15.391694 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.391670 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h6k29" event={"ID":"55d47003-e88d-472b-b137-e7cc993956ac","Type":"ContainerStarted","Data":"6f5989cfe60280797fa9c491b5ab9754b114599a2277c3616a9d99c26f2bbbf2"} Jul 21 17:02:15.821819 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.821785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:15.821976 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:15.821865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:15.822046 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.822027 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:15.822139 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.822123 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs podName:c326346e-e5d3-4618-ab4c-9b27c8946dc2 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:17.822076874 +0000 UTC m=+5.936223840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs") pod "network-metrics-daemon-9tfrw" (UID: "c326346e-e5d3-4618-ab4c-9b27c8946dc2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:15.822506 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.822488 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:15.822564 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:15.822539 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret podName:4835f1ed-ad53-45e6-b120-b08aa18e8baf nodeName:}" failed. No retries permitted until 2026-07-21 17:02:17.822525011 +0000 UTC m=+5.936671988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret") pod "global-pull-secret-syncer-6xnc7" (UID: "4835f1ed-ad53-45e6-b120-b08aa18e8baf") : object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:16.024297 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:16.023678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:16.024297 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:16.023880 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 17:02:16.024297 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:16.023990 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 17:02:16.024297 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:16.024007 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nhwpk for pod openshift-network-diagnostics/network-check-target-mq5p8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:16.024297 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:16.024066 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk podName:aee0bcc3-658e-400b-8911-829d8af34990 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:18.024047902 +0000 UTC m=+6.138194867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhwpk" (UniqueName: "kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk") pod "network-check-target-mq5p8" (UID: "aee0bcc3-658e-400b-8911-829d8af34990") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:16.245476 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:16.245418 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 16:57:14 +0000 UTC" deadline="2027-12-20 07:25:33.224992647 +0000 UTC" Jul 21 17:02:16.245476 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:16.245450 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12398h23m16.979547589s" Jul 21 17:02:16.371849 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:16.371824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:16.372000 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:16.371952 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:16.397684 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:16.397651 2562 generic.go:356] "Generic (PLEG): container finished" podID="899bf032b8c68fb92d0272de53c47dc5" containerID="090d5f7e9a2538f492504bc0143e9f84cb29b9f3441c05a3cc2a361a4864d184" exitCode=0 Jul 21 17:02:16.398005 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:16.397943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" event={"ID":"899bf032b8c68fb92d0272de53c47dc5","Type":"ContainerDied","Data":"090d5f7e9a2538f492504bc0143e9f84cb29b9f3441c05a3cc2a361a4864d184"} Jul 21 17:02:16.403224 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:16.403202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" event={"ID":"4d8c58923675c06522e424156ee02f34","Type":"ContainerStarted","Data":"52a625b252ed495cdb7aba6a6f50b62a1655748cbdacc6fc92f4721b49a4f7ac"} Jul 21 17:02:17.372560 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:17.372518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:17.373042 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:17.372659 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:17.373217 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:17.373071 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:17.373217 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:17.373199 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:17.841764 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:17.841681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:17.841764 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:17.841757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:17.841985 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:17.841888 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:17.841985 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:17.841945 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs podName:c326346e-e5d3-4618-ab4c-9b27c8946dc2 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:21.841926693 +0000 UTC m=+9.956073662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs") pod "network-metrics-daemon-9tfrw" (UID: "c326346e-e5d3-4618-ab4c-9b27c8946dc2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:17.842093 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:17.842005 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:17.842093 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:17.842037 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret podName:4835f1ed-ad53-45e6-b120-b08aa18e8baf nodeName:}" failed. No retries permitted until 2026-07-21 17:02:21.842026238 +0000 UTC m=+9.956173204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret") pod "global-pull-secret-syncer-6xnc7" (UID: "4835f1ed-ad53-45e6-b120-b08aa18e8baf") : object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:18.043044 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:18.043009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:18.043243 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:18.043197 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 17:02:18.043243 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:18.043218 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 17:02:18.043243 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:18.043231 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nhwpk for pod openshift-network-diagnostics/network-check-target-mq5p8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:18.043405 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:18.043285 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk podName:aee0bcc3-658e-400b-8911-829d8af34990 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:22.043266546 +0000 UTC m=+10.157413515 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhwpk" (UniqueName: "kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk") pod "network-check-target-mq5p8" (UID: "aee0bcc3-658e-400b-8911-829d8af34990") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:18.374279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:18.374247 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:18.374744 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:18.374380 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:19.372924 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:19.372319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:19.372924 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:19.372447 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:19.372924 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:19.372810 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:19.372924 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:19.372905 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:20.373004 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:20.372436 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:20.373004 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:20.372831 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:21.372441 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:21.372318 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:21.372441 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:21.372362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:21.372652 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:21.372463 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:21.372712 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:21.372651 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:21.874695 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:21.874647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:21.875157 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:21.874728 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:21.875157 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:21.874873 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:21.875157 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:21.874881 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:21.875157 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:21.874943 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret podName:4835f1ed-ad53-45e6-b120-b08aa18e8baf nodeName:}" failed. No retries permitted until 2026-07-21 17:02:29.874924847 +0000 UTC m=+17.989071814 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret") pod "global-pull-secret-syncer-6xnc7" (UID: "4835f1ed-ad53-45e6-b120-b08aa18e8baf") : object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:21.875157 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:21.874958 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs podName:c326346e-e5d3-4618-ab4c-9b27c8946dc2 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:29.874951216 +0000 UTC m=+17.989098180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs") pod "network-metrics-daemon-9tfrw" (UID: "c326346e-e5d3-4618-ab4c-9b27c8946dc2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:22.076876 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:22.076836 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:22.077047 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:22.077004 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 17:02:22.077047 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:22.077026 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 17:02:22.077047 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:22.077040 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nhwpk for pod openshift-network-diagnostics/network-check-target-mq5p8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:22.077211 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:22.077095 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk podName:aee0bcc3-658e-400b-8911-829d8af34990 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:30.077077753 +0000 UTC m=+18.191224732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhwpk" (UniqueName: "kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk") pod "network-check-target-mq5p8" (UID: "aee0bcc3-658e-400b-8911-829d8af34990") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:22.373512 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:22.373432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:22.373682 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:22.373550 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:23.371861 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:23.371828 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:23.372376 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:23.371872 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:23.372376 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:23.371941 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:23.372376 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:23.372130 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:24.372834 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:24.372521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:24.372834 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:24.372656 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:25.371930 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:25.371905 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:25.372003 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:25.371947 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:25.372042 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:25.372018 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:25.372153 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:25.372134 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:26.371863 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:26.371833 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:26.372348 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:26.371965 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:27.372046 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:27.372019 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:27.372476 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:27.372155 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:27.372476 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:27.372228 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:27.372476 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:27.372353 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:28.372433 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:28.372398 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:28.372869 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:28.372536 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:29.371896 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:29.371854 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:29.371896 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:29.371886 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:29.372189 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:29.371976 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:29.372189 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:29.372141 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:29.938613 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:29.938513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:29.938613 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:29.938586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:29.939070 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:29.938623 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:29.939070 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:29.938696 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:29.939070 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:29.938699 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret podName:4835f1ed-ad53-45e6-b120-b08aa18e8baf nodeName:}" failed. No retries permitted until 2026-07-21 17:02:45.938677267 +0000 UTC m=+34.052824242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret") pod "global-pull-secret-syncer-6xnc7" (UID: "4835f1ed-ad53-45e6-b120-b08aa18e8baf") : object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:29.939070 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:29.938777 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs podName:c326346e-e5d3-4618-ab4c-9b27c8946dc2 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:45.938746694 +0000 UTC m=+34.052893663 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs") pod "network-metrics-daemon-9tfrw" (UID: "c326346e-e5d3-4618-ab4c-9b27c8946dc2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:30.140734 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:30.140699 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:30.140893 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:30.140839 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 17:02:30.140893 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:30.140863 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 17:02:30.140893 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:30.140877 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nhwpk for pod openshift-network-diagnostics/network-check-target-mq5p8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:30.141026 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:30.140938 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk podName:aee0bcc3-658e-400b-8911-829d8af34990 nodeName:}" failed. No retries permitted until 2026-07-21 17:02:46.140920758 +0000 UTC m=+34.255067720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhwpk" (UniqueName: "kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk") pod "network-check-target-mq5p8" (UID: "aee0bcc3-658e-400b-8911-829d8af34990") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:30.372331 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:30.372289 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:30.372493 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:30.372421 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:31.372143 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:31.372113 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:31.372620 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:31.372153 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:31.372620 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:31.372231 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:31.372620 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:31.372388 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:32.373396 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:32.373367 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:32.373856 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:32.373484 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:33.371892 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:33.371865 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:33.372034 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:33.371904 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:33.372034 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:33.371988 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:33.372157 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:33.372078 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:34.371878 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.371727 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:34.372278 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:34.371961 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:34.442250 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.442054 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" event={"ID":"899bf032b8c68fb92d0272de53c47dc5","Type":"ContainerStarted","Data":"da16468d070eecbfc204fd8d6687095ec34dfbc55637f29c80c16adf2de8a75e"} Jul 21 17:02:34.443563 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.443543 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5vkhj" event={"ID":"5cfa7c62-ef41-4665-9794-751565d61390","Type":"ContainerStarted","Data":"e9053c00c2243ac8aa3acb5d0b2d5e687d8c5b360d266be43c945e7451f1815a"} Jul 21 17:02:34.444900 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.444877 2562 generic.go:356] "Generic (PLEG): container finished" podID="dfb9878d-5a06-4969-82ea-8bb9aee29f7c" containerID="0a5bad1dc38799f6762ad18221a72acd4f6bd5326bd7f528745ca0ef2590ae81" exitCode=0 Jul 21 17:02:34.444992 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.444956 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerDied","Data":"0a5bad1dc38799f6762ad18221a72acd4f6bd5326bd7f528745ca0ef2590ae81"} Jul 21 17:02:34.446388 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.446367 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" event={"ID":"66b23d89-c2ad-4532-a99e-ad1eee9e3905","Type":"ContainerStarted","Data":"b0b50b86c07089938e2010a4daecc3dee1db96ce28b8e462075f78b90c0b5f06"} Jul 21 17:02:34.447721 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.447702 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h6k29" event={"ID":"55d47003-e88d-472b-b137-e7cc993956ac","Type":"ContainerStarted","Data":"75fc151183b74a09954cb2599e81d9cee9907a71604223e2780eaf7c477c5509"} Jul 21 17:02:34.449028 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.449008 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwbq2" event={"ID":"3a6b668f-679a-4dc5-9fae-2e3cf8c109d4","Type":"ContainerStarted","Data":"d4583e26d07e0db12ebb342cc9426f11b2fb4d3added29e63fb7e239e4bba2b3"} Jul 21 17:02:34.450352 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.450318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rqjbb" event={"ID":"0d18ee3a-5b4e-4448-9c21-5c2a535dd38a","Type":"ContainerStarted","Data":"5947dfc6a4c0ea5ecf65ab6f8bf5742a76767ddb332a86a857d59a4518d56852"} Jul 21 17:02:34.455193 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.455167 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-clg67" event={"ID":"41c19cc9-d39e-424f-9c99-28f3d81a8522","Type":"ContainerStarted","Data":"3505c6fced5b86e1ef8ecf338c52ee32e4489c4800af8fee2b2892e28ad98e97"} Jul 21 17:02:34.456456 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.456437 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"65a09977a9d712fcf2638549558333b800d35ee236b9b12dbfe39d70b36d4b56"} Jul 21 17:02:34.465251 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.465212 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-97.ec2.internal" podStartSLOduration=21.465194149 podStartE2EDuration="21.465194149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:16.435991197 +0000 UTC m=+4.550138183" watchObservedRunningTime="2026-07-21 17:02:34.465194149 +0000 UTC m=+22.579341133" Jul 21 17:02:34.486622 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.486583 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-97.ec2.internal" podStartSLOduration=21.486569707 podStartE2EDuration="21.486569707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:34.46472734 +0000 UTC m=+22.578874326" watchObservedRunningTime="2026-07-21 17:02:34.486569707 +0000 UTC m=+22.600716693" Jul 21 17:02:34.511900 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.511861 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h6k29" podStartSLOduration=3.223869897 podStartE2EDuration="22.511849215s" totalImagesPullingTime="19.287979318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:34.486505762 +0000 UTC m=+22.600652746" watchObservedRunningTime="2026-07-21 17:02:34.511849215 +0000 UTC m=+22.625996201" Jul 21 17:02:34.527165 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.527132 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-clg67" podStartSLOduration=3.719869272 podStartE2EDuration="22.527119976s" totalImagesPullingTime="18.807250704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:34.526912621 +0000 UTC m=+22.641059606" watchObservedRunningTime="2026-07-21 17:02:34.527119976 +0000 UTC m=+22.641266952" Jul 21 17:02:34.547293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.547263 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5vkhj" podStartSLOduration=3.2624851440000002 podStartE2EDuration="22.547251488s" totalImagesPullingTime="19.284766344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:34.547119472 +0000 UTC m=+22.661266463" watchObservedRunningTime="2026-07-21 17:02:34.547251488 +0000 UTC m=+22.661398473" Jul 21 17:02:34.611862 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.611821 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rqjbb" podStartSLOduration=3.301909799 podStartE2EDuration="22.611807687s" totalImagesPullingTime="19.309897888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:34.611198041 +0000 UTC m=+22.725345026" watchObservedRunningTime="2026-07-21 17:02:34.611807687 +0000 UTC m=+22.725954672" Jul 21 17:02:34.719960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.719912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:34.826543 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:34.826520 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:35.371962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.371895 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:35.372596 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.371895 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:35.372596 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:35.372030 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:35.372596 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:35.372069 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:35.399776 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.399745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:35.459804 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.459773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-68crx" event={"ID":"3af9ba18-eaaa-4c45-815b-3a3139cb4d3f","Type":"ContainerStarted","Data":"a620941024c1b7e9360278f9c715d8c187b769c695d92790587363a6fe425358"} Jul 21 17:02:35.462611 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.462590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"c190b91d4a6dd765782ca2f6a08f7ff1e3158e2c9dc8bf21d973e8b89cb1c164"} Jul 21 17:02:35.462717 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.462646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"d990c156d80d3099fb1baeb757d350cf09f6dbc89b9074a9d1a790739eba44db"} Jul 21 17:02:35.462717 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.462680 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"71a41c60581b7f5392cfd2c95df9f428aa3b94ad468db6db3fbbbf3a427e3bfa"} Jul 21 17:02:35.462717 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.462707 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"59af6ff03c5da769dcd7593a4723c66a6fc2c5b38f58db05c3ebf539a6cf3b90"} Jul 21 17:02:35.462813 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.462734 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"dbea0c8d3d995a76eaa1b8d55a12c8f9e4a5dc2dab425a3e88be984c448e65cf"} Jul 21 17:02:35.479256 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.479221 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qwbq2" podStartSLOduration=4.7025449009999996 podStartE2EDuration="23.479208322s" totalImagesPullingTime="18.776663421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:34.643351101 +0000 UTC m=+22.757498084" watchObservedRunningTime="2026-07-21 17:02:35.479208322 +0000 UTC m=+23.593355306" Jul 21 17:02:35.790681 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.790659 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 17:02:35.803636 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:35.803615 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:36.259755 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.259671 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T17:02:35.790676619Z","UUID":"daa845a7-f3b0-405f-b4a1-96414588a4c2","Handler":null,"Name":"","Endpoint":""} Jul 21 17:02:36.261086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.261067 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 17:02:36.261208 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.261114 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 17:02:36.372518 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.372493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:36.372942 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:36.372586 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:36.418268 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.418239 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:36.467698 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.467602 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" event={"ID":"66b23d89-c2ad-4532-a99e-ad1eee9e3905","Type":"ContainerStarted","Data":"e7021cc317f530046dcf4369b6e2eb63af22dd451469fd44cad1bd6150eb48f3"} Jul 21 17:02:36.797416 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.797378 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:36.798118 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.798038 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:36.809188 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.809164 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:36.821892 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.821847 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-68crx" podStartSLOduration=5.547819152 podStartE2EDuration="24.821828874s" totalImagesPullingTime="19.274009722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:35.479482434 +0000 UTC m=+23.593629420" watchObservedRunningTime="2026-07-21 17:02:36.821828874 +0000 UTC m=+24.935975860" Jul 21 17:02:36.920631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:36.920606 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:37.054792 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.054718 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:37.168430 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.168399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:37.372665 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.372524 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:37.372992 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.372533 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:37.372992 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:37.372758 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:37.372992 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:37.372879 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:37.474376 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.474348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"4cb838c4a255374be2fb688800529b906a5bc5b2a896f1f512470f148e8842c0"} Jul 21 17:02:37.476859 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.476819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" event={"ID":"66b23d89-c2ad-4532-a99e-ad1eee9e3905","Type":"ContainerStarted","Data":"32d5ae7603eb88df04a13e3d41ee6328849d2885e04e55563543aca93d5fffbd"} Jul 21 17:02:37.477358 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.477309 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:37.477552 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.477535 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwbq2" Jul 21 17:02:37.502378 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.502337 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mkm96" podStartSLOduration=3.2526289999999998 podStartE2EDuration="25.502323833s" totalImagesPullingTime="22.249694833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:37.501570691 +0000 UTC m=+25.615717676" watchObservedRunningTime="2026-07-21 17:02:37.502323833 +0000 UTC m=+25.616470811" Jul 21 17:02:37.603852 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.603780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:37.806832 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:37.806799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:38.017083 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:38.015560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:38.245634 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:38.245592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:38.371830 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:38.371762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:38.371972 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:38.371854 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:38.440566 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:38.440535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:38.605761 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:38.605731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:39.315461 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:39.315433 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:39.371945 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:39.371911 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:39.371945 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:39.371926 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:39.372227 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:39.372025 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:39.372227 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:39.372193 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:39.415744 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:39.415710 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:39.608921 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:39.608831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:40.003679 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:40.003637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:40.203455 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:40.203397 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:40.329464 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:40.329376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:40.372559 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:40.372529 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:40.372732 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:40.372671 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:40.592662 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:40.592571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:40.912082 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:40.911998 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:41.008561 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.008532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:41.372794 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.372600 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:41.372975 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.372640 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:41.372975 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:41.372880 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:41.372975 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:41.372948 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:41.395877 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.395844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:41.488320 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.488280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" event={"ID":"fc445252-4547-4ccf-961f-4f9d50accba2","Type":"ContainerStarted","Data":"0245f4fb30e89d32b74d694e9829ed7db67fe1bd9f638444fb1bc49a7011c0e7"} Jul 21 17:02:41.488545 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.488525 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:41.490095 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.490059 2562 generic.go:356] "Generic (PLEG): container finished" podID="dfb9878d-5a06-4969-82ea-8bb9aee29f7c" containerID="5a643180845de519de1ae4f8264cab6df1ee7e1a2a0267e5bf749e2686221277" exitCode=0 Jul 21 17:02:41.490225 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.490123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerDied","Data":"5a643180845de519de1ae4f8264cab6df1ee7e1a2a0267e5bf749e2686221277"} Jul 21 17:02:41.504329 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.504303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:41.527038 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.526989 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" podStartSLOduration=9.791727346 podStartE2EDuration="29.526974179s" totalImagesPullingTime="19.735246833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:41.524346904 +0000 UTC m=+29.638493890" watchObservedRunningTime="2026-07-21 17:02:41.526974179 +0000 UTC m=+29.641121165" Jul 21 17:02:41.799804 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:41.799731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:42.373370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:42.373338 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:42.373912 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:42.373423 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:42.492534 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:42.492499 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:42.492721 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:42.492596 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:42.510402 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:42.510371 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:02:42.722972 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:42.722940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:42.830866 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:42.830832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:43.225966 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:43.225935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:43.372461 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:43.372427 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:43.372461 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:43.372455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:43.372669 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:43.372544 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:43.372716 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:43.372666 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:43.496589 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:43.496555 2562 generic.go:356] "Generic (PLEG): container finished" podID="dfb9878d-5a06-4969-82ea-8bb9aee29f7c" containerID="ca0af75141ec557c93cb35786d6db9e8d25138e47a9dcf3f0e8042d815c5ea96" exitCode=0 Jul 21 17:02:43.497018 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:43.496643 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerDied","Data":"ca0af75141ec557c93cb35786d6db9e8d25138e47a9dcf3f0e8042d815c5ea96"} Jul 21 17:02:43.597355 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:43.597325 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:43.810622 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:43.810544 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:44.319012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:44.318972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:44.371912 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:44.371880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:44.372065 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:44.372020 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:44.604316 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:44.604229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:44.870738 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:44.870632 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:45.372270 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:45.372235 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:45.372270 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:45.372267 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:45.372486 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:45.372343 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:45.372486 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:45.372464 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:45.503560 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:45.503522 2562 generic.go:356] "Generic (PLEG): container finished" podID="dfb9878d-5a06-4969-82ea-8bb9aee29f7c" containerID="bc3899998b5cca64ddc44f3b963a009f7c9eaf61799400dba28494a1629f4b8a" exitCode=0 Jul 21 17:02:45.503754 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:45.503575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerDied","Data":"bc3899998b5cca64ddc44f3b963a009f7c9eaf61799400dba28494a1629f4b8a"} Jul 21 17:02:45.577037 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:45.577002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:45.958867 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:45.958829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:45.959300 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:45.958884 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:45.959300 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:45.958972 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:45.959300 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:45.958975 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:45.959300 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:45.959022 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret podName:4835f1ed-ad53-45e6-b120-b08aa18e8baf nodeName:}" failed. No retries permitted until 2026-07-21 17:03:17.959008878 +0000 UTC m=+66.073155841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret") pod "global-pull-secret-syncer-6xnc7" (UID: "4835f1ed-ad53-45e6-b120-b08aa18e8baf") : object "kube-system"/"original-pull-secret" not registered Jul 21 17:02:45.959300 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:45.959035 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs podName:c326346e-e5d3-4618-ab4c-9b27c8946dc2 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:17.959028828 +0000 UTC m=+66.073175791 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs") pod "network-metrics-daemon-9tfrw" (UID: "c326346e-e5d3-4618-ab4c-9b27c8946dc2") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 17:02:46.160898 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:46.160859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:46.161079 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:46.161035 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 17:02:46.161079 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:46.161064 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 17:02:46.161079 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:46.161079 2562 projected.go:196] Error preparing data for projected volume kube-api-access-nhwpk for pod openshift-network-diagnostics/network-check-target-mq5p8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:46.161234 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:46.161166 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk podName:aee0bcc3-658e-400b-8911-829d8af34990 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:18.161145388 +0000 UTC m=+66.275292368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhwpk" (UniqueName: "kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk") pod "network-check-target-mq5p8" (UID: "aee0bcc3-658e-400b-8911-829d8af34990") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 17:02:46.371923 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:46.371890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:46.372118 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:46.372014 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:47.372010 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:47.371972 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:47.372475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:47.371991 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:47.372475 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:47.372127 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:47.372475 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:47.372162 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:47.425093 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:47.425042 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:47.523017 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:47.522980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:47.708395 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:47.708310 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:48.372628 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:48.372597 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:48.373072 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:48.372705 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:48.577387 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:48.577349 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:48.661985 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:48.661860 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:49.372363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:49.372319 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:49.372363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:49.372351 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:49.372620 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:49.372456 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:49.372620 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:49.372603 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:49.392662 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:49.392620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:49.602371 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:49.602330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:49.745457 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:49.745413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:49.844997 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:49.844960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:50.371923 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:50.371894 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:50.372088 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:50.372015 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:50.519062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:50.519022 2562 generic.go:356] "Generic (PLEG): container finished" podID="dfb9878d-5a06-4969-82ea-8bb9aee29f7c" containerID="7502c750ee5d4e688fd919b049f57a3e355e12c7865c76064b4ef3247b9ef013" exitCode=0 Jul 21 17:02:50.519587 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:50.519075 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerDied","Data":"7502c750ee5d4e688fd919b049f57a3e355e12c7865c76064b4ef3247b9ef013"} Jul 21 17:02:51.372226 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:51.372191 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:51.372226 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:51.372214 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:51.372451 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:51.372316 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:51.372451 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:51.372418 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:51.524395 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:51.524364 2562 generic.go:356] "Generic (PLEG): container finished" podID="dfb9878d-5a06-4969-82ea-8bb9aee29f7c" containerID="a86bfc9c1fc8f724609be8b5a281ab9b1ba6bb7ced3e18066abb2131830ebaa5" exitCode=0 Jul 21 17:02:51.524774 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:51.524426 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerDied","Data":"a86bfc9c1fc8f724609be8b5a281ab9b1ba6bb7ced3e18066abb2131830ebaa5"} Jul 21 17:02:52.080218 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:52.080174 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:52.139558 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:52.139521 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:52.372654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:52.372393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:52.372836 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:52.372680 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:52.530021 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:52.529988 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" event={"ID":"dfb9878d-5a06-4969-82ea-8bb9aee29f7c","Type":"ContainerStarted","Data":"517e92e044a146a9cf3ed0eef5171d938e56b8a65f799f728ceed0da8fb4f7e3"} Jul 21 17:02:52.557314 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:52.557224 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bxpxv" podStartSLOduration=6.733320611 podStartE2EDuration="40.557205335s" totalImagesPullingTime="33.823884724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:02:52.55672015 +0000 UTC m=+40.670867134" watchObservedRunningTime="2026-07-21 17:02:52.557205335 +0000 UTC m=+40.671352321" Jul 21 17:02:52.910515 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:52.910440 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:53.011542 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:53.011507 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:53.372331 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:53.372293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:53.372520 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:53.372292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:53.372520 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:53.372411 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:53.372520 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:53.372465 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:54.172273 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:54.172220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:54.283821 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:54.283786 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:54.372237 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:54.372203 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:54.372431 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:54.372320 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:54.490592 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:54.490550 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:54.591960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:54.591923 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:55.142506 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.142474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:55.226890 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.226860 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:55.372048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.372023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:55.372192 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:55.372164 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:55.372253 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.372216 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:55.372345 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:55.372326 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:55.584448 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.584408 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9tfrw"] Jul 21 17:02:55.584604 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.584539 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:55.584652 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:55.584632 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:55.587869 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.587823 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xnc7"] Jul 21 17:02:55.588060 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.587963 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:55.588135 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:55.588082 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:55.588635 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.588610 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mq5p8"] Jul 21 17:02:55.588766 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.588713 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:55.588834 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:55.588813 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:55.591606 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.591574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:55.663514 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:55.663479 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:57.371807 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:57.371770 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:57.371807 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:57.371808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:57.372418 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:57.371811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:57.372418 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:57.371894 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:57.372418 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:57.371961 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:57.372418 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:57.372047 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:58.622865 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:58.622831 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:58.722770 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:58.722737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:58.826340 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:58.826302 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:02:59.372378 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.372347 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:02:59.372378 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.372380 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:02:59.372596 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.372446 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:02:59.372596 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:59.372463 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6xnc7" podUID="4835f1ed-ad53-45e6-b120-b08aa18e8baf" Jul 21 17:02:59.372596 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:59.372524 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9tfrw" podUID="c326346e-e5d3-4618-ab4c-9b27c8946dc2" Jul 21 17:02:59.372596 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:02:59.372579 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mq5p8" podUID="aee0bcc3-658e-400b-8911-829d8af34990" Jul 21 17:02:59.609545 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.609517 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-97.ec2.internal" event="NodeReady" Jul 21 17:02:59.609720 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.609651 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 17:02:59.664591 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.664508 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ht659"] Jul 21 17:02:59.683164 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.683134 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nxvvr"] Jul 21 17:02:59.683317 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.683229 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.685931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.685905 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bhhmq\"" Jul 21 17:02:59.686396 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.686374 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 17:02:59.686559 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.686543 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 17:02:59.697173 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.697146 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ht659"] Jul 21 17:02:59.697320 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.697183 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nxvvr"] Jul 21 17:02:59.697320 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.697198 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rqs45"] Jul 21 17:02:59.697320 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.697195 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:02:59.699705 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.699680 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 17:02:59.699992 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.699975 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 17:02:59.700135 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.700116 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4j6ls\"" Jul 21 17:02:59.700220 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.700159 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 17:02:59.716145 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.716093 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rqs45"] Jul 21 17:02:59.716477 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.716399 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.719147 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.719124 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntxgg\"" Jul 21 17:02:59.719631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.719608 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 17:02:59.719631 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.719619 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 17:02:59.719818 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.719608 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 17:02:59.719818 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.719619 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 17:02:59.758415 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758373 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fd88f93-842d-46a9-92cf-15eda9be95d2-cert\") pod \"ingress-canary-nxvvr\" (UID: \"4fd88f93-842d-46a9-92cf-15eda9be95d2\") " pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:02:59.758415 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbzd9\" (UniqueName: \"kubernetes.io/projected/4fd88f93-842d-46a9-92cf-15eda9be95d2-kube-api-access-dbzd9\") pod \"ingress-canary-nxvvr\" (UID: \"4fd88f93-842d-46a9-92cf-15eda9be95d2\") " pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:02:59.758627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758433 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba01ebad-5daf-479a-9a36-728feef6a3c9-data-volume\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.758627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758496 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba01ebad-5daf-479a-9a36-728feef6a3c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.758627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758529 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba01ebad-5daf-479a-9a36-728feef6a3c9-crio-socket\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.758627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26fc102-988a-459c-a457-ab5918ef2002-tmp-dir\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.758627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqg7q\" (UniqueName: \"kubernetes.io/projected/e26fc102-988a-459c-a457-ab5918ef2002-kube-api-access-xqg7q\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.758627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758601 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba01ebad-5daf-479a-9a36-728feef6a3c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.758627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e26fc102-988a-459c-a457-ab5918ef2002-config-volume\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.758833 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e26fc102-988a-459c-a457-ab5918ef2002-metrics-tls\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.758833 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.758692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv97n\" (UniqueName: \"kubernetes.io/projected/ba01ebad-5daf-479a-9a36-728feef6a3c9-kube-api-access-nv97n\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.859706 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.859669 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba01ebad-5daf-479a-9a36-728feef6a3c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.859822 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.859708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e26fc102-988a-459c-a457-ab5918ef2002-config-volume\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.859861 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.859844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e26fc102-988a-459c-a457-ab5918ef2002-metrics-tls\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.859894 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.859880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv97n\" (UniqueName: \"kubernetes.io/projected/ba01ebad-5daf-479a-9a36-728feef6a3c9-kube-api-access-nv97n\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.859937 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.859916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fd88f93-842d-46a9-92cf-15eda9be95d2-cert\") pod \"ingress-canary-nxvvr\" (UID: \"4fd88f93-842d-46a9-92cf-15eda9be95d2\") " pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:02:59.860076 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860051 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbzd9\" (UniqueName: \"kubernetes.io/projected/4fd88f93-842d-46a9-92cf-15eda9be95d2-kube-api-access-dbzd9\") pod \"ingress-canary-nxvvr\" (UID: \"4fd88f93-842d-46a9-92cf-15eda9be95d2\") " pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:02:59.860190 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba01ebad-5daf-479a-9a36-728feef6a3c9-data-volume\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.860190 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba01ebad-5daf-479a-9a36-728feef6a3c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.860301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba01ebad-5daf-479a-9a36-728feef6a3c9-crio-socket\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.860301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26fc102-988a-459c-a457-ab5918ef2002-tmp-dir\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.860301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqg7q\" (UniqueName: \"kubernetes.io/projected/e26fc102-988a-459c-a457-ab5918ef2002-kube-api-access-xqg7q\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.860442 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e26fc102-988a-459c-a457-ab5918ef2002-config-volume\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.860442 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba01ebad-5daf-479a-9a36-728feef6a3c9-crio-socket\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.860537 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860478 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba01ebad-5daf-479a-9a36-728feef6a3c9-data-volume\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.860690 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860660 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26fc102-988a-459c-a457-ab5918ef2002-tmp-dir\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.860868 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.860844 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba01ebad-5daf-479a-9a36-728feef6a3c9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.864064 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.864039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e26fc102-988a-459c-a457-ab5918ef2002-metrics-tls\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.864197 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.864039 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba01ebad-5daf-479a-9a36-728feef6a3c9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.864197 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.864091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fd88f93-842d-46a9-92cf-15eda9be95d2-cert\") pod \"ingress-canary-nxvvr\" (UID: \"4fd88f93-842d-46a9-92cf-15eda9be95d2\") " pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:02:59.870546 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.870520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqg7q\" (UniqueName: \"kubernetes.io/projected/e26fc102-988a-459c-a457-ab5918ef2002-kube-api-access-xqg7q\") pod \"dns-default-ht659\" (UID: \"e26fc102-988a-459c-a457-ab5918ef2002\") " pod="openshift-dns/dns-default-ht659" Jul 21 17:02:59.870721 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.870523 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbzd9\" (UniqueName: \"kubernetes.io/projected/4fd88f93-842d-46a9-92cf-15eda9be95d2-kube-api-access-dbzd9\") pod \"ingress-canary-nxvvr\" (UID: \"4fd88f93-842d-46a9-92cf-15eda9be95d2\") " pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:02:59.870790 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.870746 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv97n\" (UniqueName: \"kubernetes.io/projected/ba01ebad-5daf-479a-9a36-728feef6a3c9-kube-api-access-nv97n\") pod \"insights-runtime-extractor-rqs45\" (UID: \"ba01ebad-5daf-479a-9a36-728feef6a3c9\") " pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:02:59.994958 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:02:59.994920 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ht659" Jul 21 17:03:00.006138 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.006097 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nxvvr" Jul 21 17:03:00.026484 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.026459 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rqs45" Jul 21 17:03:00.243614 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.243356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nxvvr"] Jul 21 17:03:00.246056 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.246029 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ht659"] Jul 21 17:03:00.247876 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:00.247848 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd88f93_842d_46a9_92cf_15eda9be95d2.slice/crio-064fc1cf6dd654801d0b9c22bd62550334c5d95b9d5a5097eab0af4dc3c0d68c WatchSource:0}: Error finding container 064fc1cf6dd654801d0b9c22bd62550334c5d95b9d5a5097eab0af4dc3c0d68c: Status 404 returned error can't find the container with id 064fc1cf6dd654801d0b9c22bd62550334c5d95b9d5a5097eab0af4dc3c0d68c Jul 21 17:03:00.249288 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:00.249261 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode26fc102_988a_459c_a457_ab5918ef2002.slice/crio-11a3cd5243c0346ee1706eb7cc10ae9d61d3a9be0b9a056b471d2c5eb1aeaf06 WatchSource:0}: Error finding container 11a3cd5243c0346ee1706eb7cc10ae9d61d3a9be0b9a056b471d2c5eb1aeaf06: Status 404 returned error can't find the container with id 11a3cd5243c0346ee1706eb7cc10ae9d61d3a9be0b9a056b471d2c5eb1aeaf06 Jul 21 17:03:00.251998 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.251970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:00.265744 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.265709 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rqs45"] Jul 21 17:03:00.270717 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:00.270689 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba01ebad_5daf_479a_9a36_728feef6a3c9.slice/crio-b0ee2b13d860567bd5e96cfc67360a1c6f91b3731902065a9f948b609c83019d WatchSource:0}: Error finding container b0ee2b13d860567bd5e96cfc67360a1c6f91b3731902065a9f948b609c83019d: Status 404 returned error can't find the container with id b0ee2b13d860567bd5e96cfc67360a1c6f91b3731902065a9f948b609c83019d Jul 21 17:03:00.336842 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.336799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:00.547694 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.547595 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqs45" event={"ID":"ba01ebad-5daf-479a-9a36-728feef6a3c9","Type":"ContainerStarted","Data":"b0ee2b13d860567bd5e96cfc67360a1c6f91b3731902065a9f948b609c83019d"} Jul 21 17:03:00.548694 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.548665 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nxvvr" event={"ID":"4fd88f93-842d-46a9-92cf-15eda9be95d2","Type":"ContainerStarted","Data":"064fc1cf6dd654801d0b9c22bd62550334c5d95b9d5a5097eab0af4dc3c0d68c"} Jul 21 17:03:00.549806 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:00.549779 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ht659" event={"ID":"e26fc102-988a-459c-a457-ab5918ef2002","Type":"ContainerStarted","Data":"11a3cd5243c0346ee1706eb7cc10ae9d61d3a9be0b9a056b471d2c5eb1aeaf06"} Jul 21 17:03:01.371922 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.371884 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:03:01.372490 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.372360 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:03:01.372866 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.372813 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:03:01.375191 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.375168 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 17:03:01.376059 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.376035 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 17:03:01.376465 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.376292 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 17:03:01.376465 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.376318 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5l8sr\"" Jul 21 17:03:01.376465 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.376368 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-59btl\"" Jul 21 17:03:01.376465 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.376400 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 17:03:01.742141 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.742092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:01.867870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:01.867832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:03.522297 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:03.522243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:03.593040 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:03.593001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:03.907202 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:03.907112 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:03.981002 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:03.980965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:04.534373 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.534339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:04.582423 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.582388 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj"] Jul 21 17:03:04.595962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.595914 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj"] Jul 21 17:03:04.595962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.595951 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.598428 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.598401 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 17:03:04.598571 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.598457 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 17:03:04.598571 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.598415 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 17:03:04.599239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.599180 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 17:03:04.599239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.599179 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zx65x\"" Jul 21 17:03:04.599608 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.599458 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 17:03:04.696828 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.696790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.696828 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.696832 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c55c061-693b-43bf-9cc3-e469aa0eccf5-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.697066 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.696856 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lcm\" (UniqueName: \"kubernetes.io/projected/7c55c061-693b-43bf-9cc3-e469aa0eccf5-kube-api-access-f8lcm\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.697066 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.696946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.797627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.797520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.797627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.797573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c55c061-693b-43bf-9cc3-e469aa0eccf5-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.797627 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.797599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lcm\" (UniqueName: \"kubernetes.io/projected/7c55c061-693b-43bf-9cc3-e469aa0eccf5-kube-api-access-f8lcm\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.797913 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.797650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.797913 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:04.797663 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 21 17:03:04.797913 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:04.797741 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-tls podName:7c55c061-693b-43bf-9cc3-e469aa0eccf5 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:05.297718266 +0000 UTC m=+53.411865234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-lqlgj" (UID: "7c55c061-693b-43bf-9cc3-e469aa0eccf5") : secret "prometheus-operator-tls" not found Jul 21 17:03:04.798646 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.798616 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c55c061-693b-43bf-9cc3-e469aa0eccf5-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.802520 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.802484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:04.808221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:04.808187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lcm\" (UniqueName: \"kubernetes.io/projected/7c55c061-693b-43bf-9cc3-e469aa0eccf5-kube-api-access-f8lcm\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:05.014393 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:05.014359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:05.116257 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:05.116146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:05.301355 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:05.301301 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:05.304493 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:05.304457 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c55c061-693b-43bf-9cc3-e469aa0eccf5-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lqlgj\" (UID: \"7c55c061-693b-43bf-9cc3-e469aa0eccf5\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:05.507782 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:05.507521 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" Jul 21 17:03:06.676823 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:06.676488 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj"] Jul 21 17:03:06.680665 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:06.680607 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c55c061_693b_43bf_9cc3_e469aa0eccf5.slice/crio-770715360a3d8eb7df480f22c208b88ca5a9ee727e2060c23d7c010ba494c95b WatchSource:0}: Error finding container 770715360a3d8eb7df480f22c208b88ca5a9ee727e2060c23d7c010ba494c95b: Status 404 returned error can't find the container with id 770715360a3d8eb7df480f22c208b88ca5a9ee727e2060c23d7c010ba494c95b Jul 21 17:03:06.936350 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:06.936192 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:07.000457 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.000427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:07.573995 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.573905 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ht659" event={"ID":"e26fc102-988a-459c-a457-ab5918ef2002","Type":"ContainerStarted","Data":"5fa121088619a64e00c8628a2297d322ef883deb785dfe588d4ba6f6ef2536ea"} Jul 21 17:03:07.574217 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.574035 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ht659" event={"ID":"e26fc102-988a-459c-a457-ab5918ef2002","Type":"ContainerStarted","Data":"14ba56ae9862814bf96a05301c5a3bbb51a4c657cca4bdd41df8422a6d819ac5"} Jul 21 17:03:07.574923 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.574853 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ht659" Jul 21 17:03:07.577172 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.577091 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqs45" event={"ID":"ba01ebad-5daf-479a-9a36-728feef6a3c9","Type":"ContainerStarted","Data":"7a238ed6c72188029e83fd7154baa69c9568db39d6825496f176db7e2dd6d7cc"} Jul 21 17:03:07.579288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.579117 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nxvvr" event={"ID":"4fd88f93-842d-46a9-92cf-15eda9be95d2","Type":"ContainerStarted","Data":"c22a55d189e5bf5a564a308480d64685c58d6027abd6b1eb95763c13182aa947"} Jul 21 17:03:07.580533 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.580504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" event={"ID":"7c55c061-693b-43bf-9cc3-e469aa0eccf5","Type":"ContainerStarted","Data":"770715360a3d8eb7df480f22c208b88ca5a9ee727e2060c23d7c010ba494c95b"} Jul 21 17:03:07.594903 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:07.594197 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ht659" podStartSLOduration=2.325739352 podStartE2EDuration="8.594171757s" totalImagesPullingTime="6.268432405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:07.593341251 +0000 UTC m=+55.707488248" watchObservedRunningTime="2026-07-21 17:03:07.594171757 +0000 UTC m=+55.708318744" Jul 21 17:03:08.270978 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:08.270915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:08.376476 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:08.376442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:09.381147 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:09.381092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:09.442620 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:09.442580 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:10.575634 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:10.575547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:10.590752 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:10.590711 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqs45" event={"ID":"ba01ebad-5daf-479a-9a36-728feef6a3c9","Type":"ContainerStarted","Data":"6b442a0f0008ce9d325cb7754010059a90066ea12d058a73afc4ba386373d22b"} Jul 21 17:03:10.593175 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:10.593140 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" event={"ID":"7c55c061-693b-43bf-9cc3-e469aa0eccf5","Type":"ContainerStarted","Data":"b449cfd4996ad510ef2c1b2345c34d728dbc383dd9b6d565f0cf152eeccc6a7a"} Jul 21 17:03:10.593343 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:10.593204 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" event={"ID":"7c55c061-693b-43bf-9cc3-e469aa0eccf5","Type":"ContainerStarted","Data":"a81e656bf10470f33c2541adfcedb5f160831c2ec1080567a428ca727ae9c539"} Jul 21 17:03:10.611806 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:10.611754 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nxvvr" podStartSLOduration=5.342011842 podStartE2EDuration="11.611736462s" totalImagesPullingTime="6.26972462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:07.612428764 +0000 UTC m=+55.726575750" watchObservedRunningTime="2026-07-21 17:03:10.611736462 +0000 UTC m=+58.725883447" Jul 21 17:03:10.612035 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:10.612003 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lqlgj" podStartSLOduration=3.055762728 podStartE2EDuration="6.611991731s" totalImagesPullingTime="3.556229003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:10.611401187 +0000 UTC m=+58.725548171" watchObservedRunningTime="2026-07-21 17:03:10.611991731 +0000 UTC m=+58.726138722" Jul 21 17:03:12.966816 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.966772 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh"] Jul 21 17:03:12.976075 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.976046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:12.980576 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.980539 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 21 17:03:12.980742 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.980580 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 21 17:03:12.981265 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.981243 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4wn7n\"" Jul 21 17:03:12.995344 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.995302 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh"] Jul 21 17:03:12.996178 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.996150 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bhlsg"] Jul 21 17:03:12.998937 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:12.998914 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.007396 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.007367 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 17:03:13.007396 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.007388 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 17:03:13.007616 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.007388 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fks9h\"" Jul 21 17:03:13.007616 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.007367 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 17:03:13.066064 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-sys\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066260 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066072 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-metrics-client-ca\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066260 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-textfile\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066260 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066177 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.066260 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066206 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.066418 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066261 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-tls\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066418 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066289 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066418 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066418 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-root\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066418 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066386 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-477kl\" (UniqueName: \"kubernetes.io/projected/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-kube-api-access-477kl\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066617 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.066617 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066467 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-wtmp\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.066617 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.066490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9mm9\" (UniqueName: \"kubernetes.io/projected/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-kube-api-access-b9mm9\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.166849 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.166806 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-477kl\" (UniqueName: \"kubernetes.io/projected/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-kube-api-access-477kl\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.166865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.167062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.166911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-wtmp\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.166929 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9mm9\" (UniqueName: \"kubernetes.io/projected/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-kube-api-access-b9mm9\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.167062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.166991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-sys\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167035 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-metrics-client-ca\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167070 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-textfile\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-wtmp\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167126 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-sys\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-tls\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167363 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-root\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167853 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:13.167398 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 21 17:03:13.167853 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-textfile\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.167853 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:13.167447 2562 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 21 17:03:13.167853 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:13.167471 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-tls podName:82e0c873-81d6-4d5c-8ebd-1bb8d245d808 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:13.667451115 +0000 UTC m=+61.781598090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-tls") pod "node-exporter-bhlsg" (UID: "82e0c873-81d6-4d5c-8ebd-1bb8d245d808") : secret "node-exporter-tls" not found Jul 21 17:03:13.167853 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:13.167509 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-tls podName:acfb9a1c-0cbd-4236-9e4d-ec04876b8c18 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:13.667492313 +0000 UTC m=+61.781639283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-z4bnh" (UID: "acfb9a1c-0cbd-4236-9e4d-ec04876b8c18") : secret "openshift-state-metrics-tls" not found Jul 21 17:03:13.167853 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.167450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-root\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.168222 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.168202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.168288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.168227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-accelerators-collector-config\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.168288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.168260 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-metrics-client-ca\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.171069 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.171037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.171250 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.171227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.178058 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.177992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9mm9\" (UniqueName: \"kubernetes.io/projected/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-kube-api-access-b9mm9\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.178401 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.178369 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-477kl\" (UniqueName: \"kubernetes.io/projected/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-kube-api-access-477kl\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.391019 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.390982 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:13.492561 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.492510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:13.591521 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.591479 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:13.671475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.671383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.671475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.671451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-tls\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.674095 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.674060 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82e0c873-81d6-4d5c-8ebd-1bb8d245d808-node-exporter-tls\") pod \"node-exporter-bhlsg\" (UID: \"82e0c873-81d6-4d5c-8ebd-1bb8d245d808\") " pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.674318 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.674300 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/acfb9a1c-0cbd-4236-9e4d-ec04876b8c18-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-z4bnh\" (UID: \"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.693134 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.693081 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:13.886227 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.886190 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" Jul 21 17:03:13.909340 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.909311 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bhlsg" Jul 21 17:03:13.952918 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:13.952841 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:13.998303 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:13.998269 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e0c873_81d6_4d5c_8ebd_1bb8d245d808.slice/crio-2f10bf082585c147183b9b96708bc05fc02855576fa372b69f9d06af97536a48 WatchSource:0}: Error finding container 2f10bf082585c147183b9b96708bc05fc02855576fa372b69f9d06af97536a48: Status 404 returned error can't find the container with id 2f10bf082585c147183b9b96708bc05fc02855576fa372b69f9d06af97536a48 Jul 21 17:03:14.040224 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.040181 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:14.069560 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.069456 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 17:03:14.072701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.072674 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.075244 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.075190 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 21 17:03:14.075794 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.075775 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 21 17:03:14.075899 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.075820 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 21 17:03:14.076047 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.076006 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 21 17:03:14.076047 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.076042 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 21 17:03:14.076537 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.076312 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 21 17:03:14.076537 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.076376 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 21 17:03:14.076537 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.076400 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 21 17:03:14.076537 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.076376 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 21 17:03:14.076537 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.076471 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wpqbt\"" Jul 21 17:03:14.093177 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.093129 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 17:03:14.120622 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.120590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:14.142733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.142698 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh"] Jul 21 17:03:14.145943 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:14.145919 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfb9a1c_0cbd_4236_9e4d_ec04876b8c18.slice/crio-7276a3f5e577dd83974db609bbf0b97271ec2b7e9bba90cd97353bd0415da358 WatchSource:0}: Error finding container 7276a3f5e577dd83974db609bbf0b97271ec2b7e9bba90cd97353bd0415da358: Status 404 returned error can't find the container with id 7276a3f5e577dd83974db609bbf0b97271ec2b7e9bba90cd97353bd0415da358 Jul 21 17:03:14.176688 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176538 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176688 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176583 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a84dd8ff-b113-4c48-bea5-0382fa878cb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176688 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176688 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a84dd8ff-b113-4c48-bea5-0382fa878cb3-config-out\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176701 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-web-config\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176729 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176751 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.176933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176907 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.177224 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.177224 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.176970 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.177224 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.177052 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnj78\" (UniqueName: \"kubernetes.io/projected/a84dd8ff-b113-4c48-bea5-0382fa878cb3-kube-api-access-fnj78\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.277612 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277570 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnj78\" (UniqueName: \"kubernetes.io/projected/a84dd8ff-b113-4c48-bea5-0382fa878cb3-kube-api-access-fnj78\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.277776 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277664 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.277776 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277702 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a84dd8ff-b113-4c48-bea5-0382fa878cb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.277776 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277730 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a84dd8ff-b113-4c48-bea5-0382fa878cb3-config-out\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-web-config\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277876 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277902 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277962 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.277993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278392 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.278021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278813 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.278781 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.278927 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:14.278903 2562 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 21 17:03:14.278991 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:14.278977 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-main-tls podName:a84dd8ff-b113-4c48-bea5-0382fa878cb3 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:14.778954614 +0000 UTC m=+62.893101577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "a84dd8ff-b113-4c48-bea5-0382fa878cb3") : secret "alertmanager-main-tls" not found Jul 21 17:03:14.280012 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:14.279462 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-trusted-ca-bundle podName:a84dd8ff-b113-4c48-bea5-0382fa878cb3 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:14.779442844 +0000 UTC m=+62.893589832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "a84dd8ff-b113-4c48-bea5-0382fa878cb3") : configmap references non-existent config key: ca-bundle.crt Jul 21 17:03:14.281693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.280979 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a84dd8ff-b113-4c48-bea5-0382fa878cb3-config-out\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.281693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.281005 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.281693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.281069 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.281693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.281211 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a84dd8ff-b113-4c48-bea5-0382fa878cb3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.281693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.281594 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-web-config\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.281693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.281675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.282814 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.282793 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.283150 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.283120 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-config-volume\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.283473 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.283450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.286697 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.286671 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnj78\" (UniqueName: \"kubernetes.io/projected/a84dd8ff-b113-4c48-bea5-0382fa878cb3-kube-api-access-fnj78\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.511914 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.511857 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jztcs" Jul 21 17:03:14.525319 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.525281 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:14.609364 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.609308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" event={"ID":"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18","Type":"ContainerStarted","Data":"6bc7ae19b88113cd55d870eaab01719c90f187516b67ad7b7e37536676085453"} Jul 21 17:03:14.609552 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.609385 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" event={"ID":"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18","Type":"ContainerStarted","Data":"f9683bae47c7888db4ab42a1a38814a643b9457789896b5df566feed7705fffa"} Jul 21 17:03:14.609552 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.609416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" event={"ID":"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18","Type":"ContainerStarted","Data":"7276a3f5e577dd83974db609bbf0b97271ec2b7e9bba90cd97353bd0415da358"} Jul 21 17:03:14.611366 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.611336 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rqs45" event={"ID":"ba01ebad-5daf-479a-9a36-728feef6a3c9","Type":"ContainerStarted","Data":"0846673ce1622df0771ce2fdb29e392e8c06236b9cdb79f4eba85b91953e71ee"} Jul 21 17:03:14.612518 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.612486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhlsg" event={"ID":"82e0c873-81d6-4d5c-8ebd-1bb8d245d808","Type":"ContainerStarted","Data":"2f10bf082585c147183b9b96708bc05fc02855576fa372b69f9d06af97536a48"} Jul 21 17:03:14.635198 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.635122 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rqs45" podStartSLOduration=2.044273898 podStartE2EDuration="15.635083857s" totalImagesPullingTime="13.590809959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:14.634905269 +0000 UTC m=+62.749052256" watchObservedRunningTime="2026-07-21 17:03:14.635083857 +0000 UTC m=+62.749230834" Jul 21 17:03:14.661875 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.661799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:14.776330 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.776237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:14.783441 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.783411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.783561 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.783480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.784439 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.784413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a84dd8ff-b113-4c48-bea5-0382fa878cb3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.786150 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.786128 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a84dd8ff-b113-4c48-bea5-0382fa878cb3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a84dd8ff-b113-4c48-bea5-0382fa878cb3\") " pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.985183 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.985126 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 21 17:03:14.992391 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:14.992358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:15.081015 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:15.080985 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:15.180861 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:15.180826 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 21 17:03:15.184380 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:15.184347 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84dd8ff_b113_4c48_bea5_0382fa878cb3.slice/crio-88b4100c704bb146a950e0c41ef9d0a208b1610e7ddef0593ac88844b06a90a2 WatchSource:0}: Error finding container 88b4100c704bb146a950e0c41ef9d0a208b1610e7ddef0593ac88844b06a90a2: Status 404 returned error can't find the container with id 88b4100c704bb146a950e0c41ef9d0a208b1610e7ddef0593ac88844b06a90a2 Jul 21 17:03:15.562144 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:15.562087 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:15.618666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:15.618570 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhlsg" event={"ID":"82e0c873-81d6-4d5c-8ebd-1bb8d245d808","Type":"ContainerStarted","Data":"bf43c6ed6ac963064db87c21dec8f92a52b6ccea4d19308ae215831a778ae8ae"} Jul 21 17:03:15.619987 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:15.619940 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerStarted","Data":"88b4100c704bb146a950e0c41ef9d0a208b1610e7ddef0593ac88844b06a90a2"} Jul 21 17:03:15.645088 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:15.645052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:16.072766 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.072730 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:16.128221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.128174 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-bc9b87b4-j2s65"] Jul 21 17:03:16.130821 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.130795 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.133264 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.133237 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 21 17:03:16.133264 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.133254 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 21 17:03:16.133488 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.133237 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 21 17:03:16.133488 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.133293 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-98tqv\"" Jul 21 17:03:16.133663 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.133605 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 21 17:03:16.133843 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.133715 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 21 17:03:16.133843 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.133758 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-enaq2mg0j89e8\"" Jul 21 17:03:16.146036 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.146001 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bc9b87b4-j2s65"] Jul 21 17:03:16.164853 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.164825 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:16.197469 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.197431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hncbt\" (UniqueName: \"kubernetes.io/projected/9f737a90-b272-41e2-b5a4-4c6b3be38f59-kube-api-access-hncbt\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.197469 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.197469 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-grpc-tls\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.197660 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.197498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.197660 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.197527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f737a90-b272-41e2-b5a4-4c6b3be38f59-metrics-client-ca\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.197660 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.197575 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.198251 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.198022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.198251 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.198096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.198251 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.198160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-tls\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.295091 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.295047 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:16.299224 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.299408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299233 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.299408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.299408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299315 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-tls\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.299408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299401 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncbt\" (UniqueName: \"kubernetes.io/projected/9f737a90-b272-41e2-b5a4-4c6b3be38f59-kube-api-access-hncbt\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.299592 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-grpc-tls\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.299752 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.299845 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.299768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f737a90-b272-41e2-b5a4-4c6b3be38f59-metrics-client-ca\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.300587 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.300556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f737a90-b272-41e2-b5a4-4c6b3be38f59-metrics-client-ca\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.303018 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.302598 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.303018 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.302975 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.303748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.303480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-grpc-tls\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.303748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.303672 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-tls\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.303748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.303737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.304282 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.304259 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9f737a90-b272-41e2-b5a4-4c6b3be38f59-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.309749 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.309664 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncbt\" (UniqueName: \"kubernetes.io/projected/9f737a90-b272-41e2-b5a4-4c6b3be38f59-kube-api-access-hncbt\") pod \"thanos-querier-bc9b87b4-j2s65\" (UID: \"9f737a90-b272-41e2-b5a4-4c6b3be38f59\") " pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.441423 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.441314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:16.494996 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.494955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:16.588799 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.588691 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-bc9b87b4-j2s65"] Jul 21 17:03:16.593758 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.593728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:16.594210 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:16.594171 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f737a90_b272_41e2_b5a4_4c6b3be38f59.slice/crio-b6a4bf4cb13fe6d835cce546e4f461b45738baa36f786afdeaa6cbd260d4c8ef WatchSource:0}: Error finding container b6a4bf4cb13fe6d835cce546e4f461b45738baa36f786afdeaa6cbd260d4c8ef: Status 404 returned error can't find the container with id b6a4bf4cb13fe6d835cce546e4f461b45738baa36f786afdeaa6cbd260d4c8ef Jul 21 17:03:16.625752 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.625714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" event={"ID":"9f737a90-b272-41e2-b5a4-4c6b3be38f59","Type":"ContainerStarted","Data":"b6a4bf4cb13fe6d835cce546e4f461b45738baa36f786afdeaa6cbd260d4c8ef"} Jul 21 17:03:16.628167 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.628138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" event={"ID":"acfb9a1c-0cbd-4236-9e4d-ec04876b8c18","Type":"ContainerStarted","Data":"3443af5fe55fab2c2fd4a65cc399a2510097df598672002887a27e0cdf7046a7"} Jul 21 17:03:16.629715 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.629683 2562 generic.go:356] "Generic (PLEG): container finished" podID="82e0c873-81d6-4d5c-8ebd-1bb8d245d808" containerID="bf43c6ed6ac963064db87c21dec8f92a52b6ccea4d19308ae215831a778ae8ae" exitCode=0 Jul 21 17:03:16.629867 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.629734 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhlsg" event={"ID":"82e0c873-81d6-4d5c-8ebd-1bb8d245d808","Type":"ContainerDied","Data":"bf43c6ed6ac963064db87c21dec8f92a52b6ccea4d19308ae215831a778ae8ae"} Jul 21 17:03:16.646689 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:16.646634 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-z4bnh" podStartSLOduration=2.779014748 podStartE2EDuration="4.646615669s" totalImagesPullingTime="1.867600921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:16.645706115 +0000 UTC m=+64.759853101" watchObservedRunningTime="2026-07-21 17:03:16.646615669 +0000 UTC m=+64.760762653" Jul 21 17:03:17.635400 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.635367 2562 generic.go:356] "Generic (PLEG): container finished" podID="a84dd8ff-b113-4c48-bea5-0382fa878cb3" containerID="97abca7af59fd3f28c20606bc4c211a7f8b1214e4417f130f5d14c9ccfb32b55" exitCode=0 Jul 21 17:03:17.635921 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.635460 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerDied","Data":"97abca7af59fd3f28c20606bc4c211a7f8b1214e4417f130f5d14c9ccfb32b55"} Jul 21 17:03:17.637734 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.637710 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhlsg" event={"ID":"82e0c873-81d6-4d5c-8ebd-1bb8d245d808","Type":"ContainerStarted","Data":"6c428148cf984bc960f5ec39bb25a775aad452cc8503dfa103f84e2cb10e4a30"} Jul 21 17:03:17.637815 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.637762 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bhlsg" event={"ID":"82e0c873-81d6-4d5c-8ebd-1bb8d245d808","Type":"ContainerStarted","Data":"6f7dfaa93ae5347214684b182388c933e12834a6b0af775f0f49e2cd72e390e7"} Jul 21 17:03:17.684402 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.684342 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bhlsg" podStartSLOduration=4.1672896680000004 podStartE2EDuration="5.684320757s" totalImagesPullingTime="1.40049406s" totalInitContainerRuntime="116.537029ms" isStatefulPod=true podCreationTimestamp="2026-07-21 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:17.683839782 +0000 UTC m=+65.797986769" watchObservedRunningTime="2026-07-21 17:03:17.684320757 +0000 UTC m=+65.798467743" Jul 21 17:03:17.714888 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.714859 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:17.734752 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.734720 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9"] Jul 21 17:03:17.736781 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.736761 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:17.739213 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.739190 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 21 17:03:17.739338 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.739219 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-zht7j\"" Jul 21 17:03:17.747751 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.747719 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9"] Jul 21 17:03:17.814829 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.814793 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb6qz\" (UniqueName: \"kubernetes.io/projected/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-kube-api-access-mb6qz\") pod \"monitoring-plugin-7b7b7bc8bf-vvnz9\" (UID: \"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:17.815017 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.814851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-vvnz9\" (UID: \"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:17.836520 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.836492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:17.916254 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.916147 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb6qz\" (UniqueName: \"kubernetes.io/projected/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-kube-api-access-mb6qz\") pod \"monitoring-plugin-7b7b7bc8bf-vvnz9\" (UID: \"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:17.916254 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.916210 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-vvnz9\" (UID: \"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:17.916501 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:17.916367 2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 21 17:03:17.916501 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:03:17.916436 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-monitoring-plugin-cert podName:5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0 nodeName:}" failed. No retries permitted until 2026-07-21 17:03:18.416417429 +0000 UTC m=+66.530564391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-vvnz9" (UID: "5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0") : secret "monitoring-plugin-cert" not found Jul 21 17:03:17.926178 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:17.926147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb6qz\" (UniqueName: \"kubernetes.io/projected/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-kube-api-access-mb6qz\") pod \"monitoring-plugin-7b7b7bc8bf-vvnz9\" (UID: \"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:18.017360 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.017319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:03:18.017569 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.017432 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:03:18.020057 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.020033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 17:03:18.020192 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.020144 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 17:03:18.030313 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.030285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c326346e-e5d3-4618-ab4c-9b27c8946dc2-metrics-certs\") pod \"network-metrics-daemon-9tfrw\" (UID: \"c326346e-e5d3-4618-ab4c-9b27c8946dc2\") " pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:03:18.030451 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.030339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4835f1ed-ad53-45e6-b120-b08aa18e8baf-original-pull-secret\") pod \"global-pull-secret-syncer-6xnc7\" (UID: \"4835f1ed-ad53-45e6-b120-b08aa18e8baf\") " pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:03:18.134973 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.134896 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:18.200873 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.200785 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5l8sr\"" Jul 21 17:03:18.202712 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.202681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:18.208872 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.208839 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9tfrw" Jul 21 17:03:18.209022 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.208901 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6xnc7" Jul 21 17:03:18.219249 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.219202 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:03:18.221872 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.221830 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 17:03:18.231821 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.231787 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 17:03:18.243284 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.243253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwpk\" (UniqueName: \"kubernetes.io/projected/aee0bcc3-658e-400b-8911-829d8af34990-kube-api-access-nhwpk\") pod \"network-check-target-mq5p8\" (UID: \"aee0bcc3-658e-400b-8911-829d8af34990\") " pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:03:18.358073 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.358040 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6xnc7"] Jul 21 17:03:18.361718 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:18.361680 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4835f1ed_ad53_45e6_b120_b08aa18e8baf.slice/crio-6395dcb06028f54783bc05579cb853f421efc9a5e7c515c39e023da758517a48 WatchSource:0}: Error finding container 6395dcb06028f54783bc05579cb853f421efc9a5e7c515c39e023da758517a48: Status 404 returned error can't find the container with id 6395dcb06028f54783bc05579cb853f421efc9a5e7c515c39e023da758517a48 Jul 21 17:03:18.378049 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.378017 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9tfrw"] Jul 21 17:03:18.381200 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:18.381165 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc326346e_e5d3_4618_ab4c_9b27c8946dc2.slice/crio-01cf6004fc551dfea01a5df133564312bc7426f875099a7c2271ba1113ffe0cc WatchSource:0}: Error finding container 01cf6004fc551dfea01a5df133564312bc7426f875099a7c2271ba1113ffe0cc: Status 404 returned error can't find the container with id 01cf6004fc551dfea01a5df133564312bc7426f875099a7c2271ba1113ffe0cc Jul 21 17:03:18.421524 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.421483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-vvnz9\" (UID: \"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:18.424212 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.424177 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-vvnz9\" (UID: \"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:18.490459 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.490426 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-59btl\"" Jul 21 17:03:18.498394 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.498357 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:03:18.588334 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.588047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ht659" Jul 21 17:03:18.645700 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.645674 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:18.647022 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.646983 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tfrw" event={"ID":"c326346e-e5d3-4618-ab4c-9b27c8946dc2","Type":"ContainerStarted","Data":"01cf6004fc551dfea01a5df133564312bc7426f875099a7c2271ba1113ffe0cc"} Jul 21 17:03:18.649316 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.649278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6xnc7" event={"ID":"4835f1ed-ad53-45e6-b120-b08aa18e8baf","Type":"ContainerStarted","Data":"6395dcb06028f54783bc05579cb853f421efc9a5e7c515c39e023da758517a48"} Jul 21 17:03:18.659897 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.659018 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mq5p8"] Jul 21 17:03:18.662542 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:18.662496 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee0bcc3_658e_400b_8911_829d8af34990.slice/crio-f39b24e848b4cade3c6cf5e574a58a45d57553831174241c2b65097959cecb7f WatchSource:0}: Error finding container f39b24e848b4cade3c6cf5e574a58a45d57553831174241c2b65097959cecb7f: Status 404 returned error can't find the container with id f39b24e848b4cade3c6cf5e574a58a45d57553831174241c2b65097959cecb7f Jul 21 17:03:18.727422 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.727383 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:18.811172 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.811140 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9"] Jul 21 17:03:18.814860 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:03:18.814819 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec0f1e7_2f84_49e2_a209_2c5cd4a318f0.slice/crio-cdcc3d32a5482d124d5adbe52b72be379e1d7929448b747d7d50a29325dccfa7 WatchSource:0}: Error finding container cdcc3d32a5482d124d5adbe52b72be379e1d7929448b747d7d50a29325dccfa7: Status 404 returned error can't find the container with id cdcc3d32a5482d124d5adbe52b72be379e1d7929448b747d7d50a29325dccfa7 Jul 21 17:03:18.815001 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.814977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:18.957680 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:18.957574 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:19.016074 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.015986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:19.182510 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.182467 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:19.288912 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.288765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:19.381927 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.379832 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 17:03:19.385380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.385342 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.388515 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.388462 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 17:03:19.388827 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.388790 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.388933 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nz5dv\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.388811 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.389215 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.389446 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.389458 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.389678 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.389725 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2g5d9msts3r0c\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.389826 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.389902 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.390165 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.390298 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.390371 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 17:03:19.390654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.390413 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 17:03:19.393824 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.393467 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 17:03:19.399158 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.398360 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 17:03:19.399470 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.399430 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:19.403092 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.402988 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439532 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439645 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439674 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/764930c1-c169-4bc9-8c14-cd3b7a5e5659-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439715 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439739 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439845 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439899 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-web-config\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.439991 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-config\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.440370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.440027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.441388 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.440054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/764930c1-c169-4bc9-8c14-cd3b7a5e5659-config-out\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.441388 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.440079 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.441388 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.440131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9ll\" (UniqueName: \"kubernetes.io/projected/764930c1-c169-4bc9-8c14-cd3b7a5e5659-kube-api-access-9w9ll\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.441388 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.440159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541054 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.540963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541054 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541067 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541134 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-web-config\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-config\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541283 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/764930c1-c169-4bc9-8c14-cd3b7a5e5659-config-out\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541310 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541304 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9ll\" (UniqueName: \"kubernetes.io/projected/764930c1-c169-4bc9-8c14-cd3b7a5e5659-kube-api-access-9w9ll\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541397 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541485 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/764930c1-c169-4bc9-8c14-cd3b7a5e5659-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541549 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.541748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.541743 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.542426 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.542396 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.542961 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.542930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.543931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.543904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.544280 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.544256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-web-config\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.544955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.545033 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.546091 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.546631 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.546687 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.546800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/764930c1-c169-4bc9-8c14-cd3b7a5e5659-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.547662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-config\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.547839 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.548351 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/764930c1-c169-4bc9-8c14-cd3b7a5e5659-config-out\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.549080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/764930c1-c169-4bc9-8c14-cd3b7a5e5659-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.549586 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.550072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.550197 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.551085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.551016 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/764930c1-c169-4bc9-8c14-cd3b7a5e5659-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.555010 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.554954 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9ll\" (UniqueName: \"kubernetes.io/projected/764930c1-c169-4bc9-8c14-cd3b7a5e5659-kube-api-access-9w9ll\") pod \"prometheus-k8s-0\" (UID: \"764930c1-c169-4bc9-8c14-cd3b7a5e5659\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.659714 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.659660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" event={"ID":"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0","Type":"ContainerStarted","Data":"cdcc3d32a5482d124d5adbe52b72be379e1d7929448b747d7d50a29325dccfa7"} Jul 21 17:03:19.666807 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.666761 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mq5p8" event={"ID":"aee0bcc3-658e-400b-8911-829d8af34990","Type":"ContainerStarted","Data":"f39b24e848b4cade3c6cf5e574a58a45d57553831174241c2b65097959cecb7f"} Jul 21 17:03:19.702764 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.702726 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:19.947851 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:19.947743 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:20.016520 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:20.016470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:20.973665 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:20.973629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 17:03:21.456812 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.456680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:21.560324 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.560225 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:21.685960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.685693 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerStarted","Data":"eb46a4bb146456a9d897597b7a4657ae5307a782bace6d3e3cfff59db1874536"} Jul 21 17:03:21.685960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.685787 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerStarted","Data":"c8c1814542655b7e3b2ba1298e8af4f63e3c632ab34f0e54cb79e584a6954019"} Jul 21 17:03:21.685960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.685830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerStarted","Data":"6df48b6d4d8b39db54cd0bae72577f12d5497fdd592700692795ab71b9448854"} Jul 21 17:03:21.685960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.685867 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerStarted","Data":"1a23ba33ab97853a206449018f0bdaab60f2c5d3c1d9519215d78c991c60be7d"} Jul 21 17:03:21.685960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.685907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerStarted","Data":"830df66dcdca4821cee01d9c3446221892b56be6cbba74adef0df7cd0351449a"} Jul 21 17:03:21.690309 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.690266 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" event={"ID":"9f737a90-b272-41e2-b5a4-4c6b3be38f59","Type":"ContainerStarted","Data":"09b1cb536bb2706cfb4246c6bd6721c5c4475edfb8413da0d940d11fecfd232a"} Jul 21 17:03:21.690492 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.690360 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" event={"ID":"9f737a90-b272-41e2-b5a4-4c6b3be38f59","Type":"ContainerStarted","Data":"49705a75ef35b5861a13cd93b45ec660096e6b9c3c4e617196e9475f6ca6aca5"} Jul 21 17:03:21.690492 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.690403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" event={"ID":"9f737a90-b272-41e2-b5a4-4c6b3be38f59","Type":"ContainerStarted","Data":"fb0aaa554a68cb7fe14ad0cf6624f79e7ddf6db9d8313e79962d867ab0f59d0c"} Jul 21 17:03:21.694128 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.693890 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tfrw" event={"ID":"c326346e-e5d3-4618-ab4c-9b27c8946dc2","Type":"ContainerStarted","Data":"7e2f3f06cb73193289103f6905bb5f69006c92566abe1106fb788925004412cd"} Jul 21 17:03:21.694128 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.693959 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9tfrw" event={"ID":"c326346e-e5d3-4618-ab4c-9b27c8946dc2","Type":"ContainerStarted","Data":"6becd1d5595e76c5c485ddcfe7fb96b208b8d29d3faccf60060306cb0d531cf2"} Jul 21 17:03:21.697334 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.697290 2562 generic.go:356] "Generic (PLEG): container finished" podID="764930c1-c169-4bc9-8c14-cd3b7a5e5659" containerID="fd2d76158cda573fef4b18c6b73956b5cf88b33a91eb21974eadcbfcf161bdf7" exitCode=0 Jul 21 17:03:21.697533 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.697364 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerDied","Data":"fd2d76158cda573fef4b18c6b73956b5cf88b33a91eb21974eadcbfcf161bdf7"} Jul 21 17:03:21.697533 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.697438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerStarted","Data":"8d3c461040f013473b5d12dff4b9b6bfdfbecd046983b7f953f00bf14b457aeb"} Jul 21 17:03:21.713564 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.713495 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9tfrw" podStartSLOduration=67.331144565 podStartE2EDuration="1m9.713470409s" totalImagesPullingTime="2.382325844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:21.711904157 +0000 UTC m=+69.826051143" watchObservedRunningTime="2026-07-21 17:03:21.713470409 +0000 UTC m=+69.827617394" Jul 21 17:03:21.816246 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.816161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:21.918231 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:21.917384 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:22.074162 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:22.074034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:23.123733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:23.123683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:24.157083 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:24.157041 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:24.260186 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:24.260151 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:25.143829 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:25.143722 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:25.244088 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:25.244055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:25.943149 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:25.943093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:26.044240 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.044178 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:26.724042 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.724007 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mq5p8" event={"ID":"aee0bcc3-658e-400b-8911-829d8af34990","Type":"ContainerStarted","Data":"e1bd05c4a044655622967cbe8d10fd5796d322f0b05c1b65ed6309ddd46ee7a2"} Jul 21 17:03:26.724526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.724205 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:03:26.727643 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.727598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerStarted","Data":"eb8331021fe71464bed48ce5d6e7c765164563cb33c8f346b102b57e2000891a"} Jul 21 17:03:26.727800 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.727672 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerStarted","Data":"ba9ec9d1ba67eab64464e53cd91f83d0ce78dbf1cec7325703cf79764ace8aed"} Jul 21 17:03:26.727800 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.727724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerStarted","Data":"06c17bb348c729836be98d7fa7bd643fa6adf42a9f19d5e98895c6da7e745937"} Jul 21 17:03:26.727800 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.727751 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerStarted","Data":"82a2182e1de8e9843a73889f658ff5bf20b3a24db394ad48b6fa711a4122d9e1"} Jul 21 17:03:26.727800 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.727774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerStarted","Data":"5982789e7a881a4c80c8d7cc688251d04e72494fc4a7e0440945c16c5731a1cb"} Jul 21 17:03:26.728017 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.727804 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"764930c1-c169-4bc9-8c14-cd3b7a5e5659","Type":"ContainerStarted","Data":"25d4abf49a9baa0a91fed08128db40be0b1281f41c31b2eebc65877731e07e51"} Jul 21 17:03:26.731029 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.730999 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a84dd8ff-b113-4c48-bea5-0382fa878cb3","Type":"ContainerStarted","Data":"e43e0b1d1c13bf3f5ead7df4d1feebc450107fbf9269ab9a0e615af26468e9a7"} Jul 21 17:03:26.734075 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.734045 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" event={"ID":"9f737a90-b272-41e2-b5a4-4c6b3be38f59","Type":"ContainerStarted","Data":"7a2b467ba59299e6bfb77816654a378bea96736fe4a6c5515524292cddb0581a"} Jul 21 17:03:26.734266 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.734159 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" event={"ID":"9f737a90-b272-41e2-b5a4-4c6b3be38f59","Type":"ContainerStarted","Data":"87abbaf1c7db96ccbffbc5037be0094da754f4b9155db86971b729af51064d89"} Jul 21 17:03:26.734266 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.734199 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" event={"ID":"9f737a90-b272-41e2-b5a4-4c6b3be38f59","Type":"ContainerStarted","Data":"f01bb28a9d876637ff3e0c79bf7f96fe6f2358bccb4368af9361a88f4a5e4693"} Jul 21 17:03:26.734380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.734331 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:26.735821 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.735788 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6xnc7" event={"ID":"4835f1ed-ad53-45e6-b120-b08aa18e8baf","Type":"ContainerStarted","Data":"13626aa227b2ac446aa10ea0b0a6d2a140666ead30dd2c133a02b18b67146311"} Jul 21 17:03:26.737492 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.737470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" event={"ID":"5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0","Type":"ContainerStarted","Data":"033e62d9f948767edf125032342fd983a4a0b159e8680adc15878f7b905d4bc6"} Jul 21 17:03:26.737693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.737674 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:26.740741 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.740716 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" Jul 21 17:03:26.742578 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.742552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" Jul 21 17:03:26.742974 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.742941 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mq5p8" podStartSLOduration=67.30902094 podStartE2EDuration="1m14.742929423s" totalImagesPullingTime="7.433908483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:26.741286319 +0000 UTC m=+74.855433304" watchObservedRunningTime="2026-07-21 17:03:26.742929423 +0000 UTC m=+74.857076408" Jul 21 17:03:26.771694 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.771618 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.106743014 podStartE2EDuration="12.771594518s" totalImagesPullingTime="9.644684953s" totalInitContainerRuntime="20.166551ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:03:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:26.7701959 +0000 UTC m=+74.884342887" watchObservedRunningTime="2026-07-21 17:03:26.771594518 +0000 UTC m=+74.885741504" Jul 21 17:03:26.795797 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.795700 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-bc9b87b4-j2s65" podStartSLOduration=1.640094143 podStartE2EDuration="10.79567752s" totalImagesPullingTime="9.155583377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:26.795026294 +0000 UTC m=+74.909173280" watchObservedRunningTime="2026-07-21 17:03:26.79567752 +0000 UTC m=+74.909824507" Jul 21 17:03:26.852020 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.851967 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.42155737 podStartE2EDuration="7.851942378s" totalImagesPullingTime="4.411919206s" totalInitContainerRuntime="18.465802ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:26.850663882 +0000 UTC m=+74.964810871" watchObservedRunningTime="2026-07-21 17:03:26.851942378 +0000 UTC m=+74.966089345" Jul 21 17:03:26.853400 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.853362 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-vvnz9" podStartSLOduration=2.571456761 podStartE2EDuration="9.85334794s" totalImagesPullingTime="7.281891179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:26.822727389 +0000 UTC m=+74.936874379" watchObservedRunningTime="2026-07-21 17:03:26.85334794 +0000 UTC m=+74.967494933" Jul 21 17:03:26.873909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:26.873855 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6xnc7" podStartSLOduration=67.13920456 podStartE2EDuration="1m14.873837793s" totalImagesPullingTime="7.734633233s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:03:26.873291523 +0000 UTC m=+74.987438510" watchObservedRunningTime="2026-07-21 17:03:26.873837793 +0000 UTC m=+74.987984777" Jul 21 17:03:29.287320 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:29.287281 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:29.386968 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:29.386933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:29.448300 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:29.448269 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:29.703386 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:29.703293 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:03:30.533397 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:30.533364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:30.604223 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:30.604187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:35.704338 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:35.704214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:35.804962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:35.804930 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:36.175977 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:36.175940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:36.274046 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:36.274014 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:36.374567 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:36.374531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:37.077697 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:37.077665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:37.185947 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:37.185916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:52.077798 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:52.077760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:52.139025 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:52.138988 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:03:57.746872 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:03:57.746837 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mq5p8" Jul 21 17:04:19.703915 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:19.703873 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:04:19.724332 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:19.724303 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:04:19.967675 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:19.967591 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 17:04:29.192693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:29.192655 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:29.268702 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:29.268665 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:41.174326 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:41.174286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:41.239597 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:41.239560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:47.708278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:47.708240 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:47.810062 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:47.810018 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:47.908625 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:47.908588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:48.621276 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:48.621243 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:48.722025 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:48.721994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:52.129183 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:52.129130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:52.232279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:52.232247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:58.449088 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:58.449048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:58.549886 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:58.549823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:04:58.616157 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:04:58.616097 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:05:30.669828 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:05:30.669797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:05:41.711023 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:05:41.710986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:05:41.803410 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:05:41.803377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:09.048471 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:09.048435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:10.639457 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:10.639426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:13.954694 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:13.954663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:14.100806 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:14.100763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:15.492598 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:15.492565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:15.620853 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:15.620814 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:15.952053 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:15.951965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:16.234831 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:16.234804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:16.395710 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:16.395671 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:17.970491 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:17.970455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:18.166401 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:18.166370 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:19.807347 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:19.807313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:19.907935 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:19.907905 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:24.570888 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:24.570854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:24.670717 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:24.670682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:25.551460 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:25.551427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:26.088138 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.088090 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fbszv"] Jul 21 17:06:26.090390 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.090371 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.094764 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.094741 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-mlrnb\"" Jul 21 17:06:26.095177 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.095161 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 17:06:26.095495 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.095482 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 17:06:26.101685 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.101660 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fbszv"] Jul 21 17:06:26.149985 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.149952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdq5n\" (UniqueName: \"kubernetes.io/projected/0128f69f-bb2a-442d-966a-203d11d7969a-kube-api-access-pdq5n\") pod \"cert-manager-55b4d954d5-fbszv\" (UID: \"0128f69f-bb2a-442d-966a-203d11d7969a\") " pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.149985 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.149989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0128f69f-bb2a-442d-966a-203d11d7969a-bound-sa-token\") pod \"cert-manager-55b4d954d5-fbszv\" (UID: \"0128f69f-bb2a-442d-966a-203d11d7969a\") " pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.250774 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.250737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdq5n\" (UniqueName: \"kubernetes.io/projected/0128f69f-bb2a-442d-966a-203d11d7969a-kube-api-access-pdq5n\") pod \"cert-manager-55b4d954d5-fbszv\" (UID: \"0128f69f-bb2a-442d-966a-203d11d7969a\") " pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.250957 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.250780 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0128f69f-bb2a-442d-966a-203d11d7969a-bound-sa-token\") pod \"cert-manager-55b4d954d5-fbszv\" (UID: \"0128f69f-bb2a-442d-966a-203d11d7969a\") " pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.262196 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.262166 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0128f69f-bb2a-442d-966a-203d11d7969a-bound-sa-token\") pod \"cert-manager-55b4d954d5-fbszv\" (UID: \"0128f69f-bb2a-442d-966a-203d11d7969a\") " pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.262325 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.262217 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdq5n\" (UniqueName: \"kubernetes.io/projected/0128f69f-bb2a-442d-966a-203d11d7969a-kube-api-access-pdq5n\") pod \"cert-manager-55b4d954d5-fbszv\" (UID: \"0128f69f-bb2a-442d-966a-203d11d7969a\") " pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.415948 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.415853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fbszv" Jul 21 17:06:26.554036 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:26.554012 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fbszv"] Jul 21 17:06:26.557346 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:06:26.557313 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0128f69f_bb2a_442d_966a_203d11d7969a.slice/crio-4039de07d2e5bba50fbaadd478eca95fcd728d0bfa6d9d2e239cde58afa05f8c WatchSource:0}: Error finding container 4039de07d2e5bba50fbaadd478eca95fcd728d0bfa6d9d2e239cde58afa05f8c: Status 404 returned error can't find the container with id 4039de07d2e5bba50fbaadd478eca95fcd728d0bfa6d9d2e239cde58afa05f8c Jul 21 17:06:27.425022 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:27.424986 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fbszv" event={"ID":"0128f69f-bb2a-442d-966a-203d11d7969a","Type":"ContainerStarted","Data":"4039de07d2e5bba50fbaadd478eca95fcd728d0bfa6d9d2e239cde58afa05f8c"} Jul 21 17:06:30.438802 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:30.438724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fbszv" event={"ID":"0128f69f-bb2a-442d-966a-203d11d7969a","Type":"ContainerStarted","Data":"9211ece90dec8bb66b68228546057b2c2509797893cc0cb610b932bc1d659a82"} Jul 21 17:06:30.455325 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:30.455286 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fbszv" podStartSLOduration=0.935418292 podStartE2EDuration="4.455270208s" totalImagesPullingTime="3.519851916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:06:30.454644156 +0000 UTC m=+258.568791140" watchObservedRunningTime="2026-07-21 17:06:30.455270208 +0000 UTC m=+258.569417193" Jul 21 17:06:30.846870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:30.846836 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:30.945235 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:30.945200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:34.955849 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:34.955769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:37.426721 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:37.426692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:37.526265 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:37.526235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:39.335309 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:39.335276 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:52.209011 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:52.208977 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:53.327157 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.327123 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk"] Jul 21 17:06:53.331482 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.331462 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.335023 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.334979 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 17:06:53.335023 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.334979 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 17:06:53.335238 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.335182 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 17:06:53.335285 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.335270 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tgbxz\"" Jul 21 17:06:53.335382 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.335361 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 17:06:53.355332 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.355310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk"] Jul 21 17:06:53.475225 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.475195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlvk\" (UniqueName: \"kubernetes.io/projected/fa65b591-bb96-4c3b-be95-cfd71f969bee-kube-api-access-wwlvk\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.475225 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.475226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa65b591-bb96-4c3b-be95-cfd71f969bee-tmp\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.475408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.475255 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa65b591-bb96-4c3b-be95-cfd71f969bee-manifests\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.475408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.475303 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa65b591-bb96-4c3b-be95-cfd71f969bee-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.475408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.475365 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa65b591-bb96-4c3b-be95-cfd71f969bee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.576707 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.576678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa65b591-bb96-4c3b-be95-cfd71f969bee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.576863 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.576720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlvk\" (UniqueName: \"kubernetes.io/projected/fa65b591-bb96-4c3b-be95-cfd71f969bee-kube-api-access-wwlvk\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.576863 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.576738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa65b591-bb96-4c3b-be95-cfd71f969bee-tmp\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.576863 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.576769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa65b591-bb96-4c3b-be95-cfd71f969bee-manifests\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.576863 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.576792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa65b591-bb96-4c3b-be95-cfd71f969bee-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.577189 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.577136 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fa65b591-bb96-4c3b-be95-cfd71f969bee-manifests\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.579012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.578983 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fa65b591-bb96-4c3b-be95-cfd71f969bee-tmp\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.579217 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.579199 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa65b591-bb96-4c3b-be95-cfd71f969bee-webhook-cert\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.579304 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.579282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fa65b591-bb96-4c3b-be95-cfd71f969bee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.588661 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.588640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlvk\" (UniqueName: \"kubernetes.io/projected/fa65b591-bb96-4c3b-be95-cfd71f969bee-kube-api-access-wwlvk\") pod \"opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk\" (UID: \"fa65b591-bb96-4c3b-be95-cfd71f969bee\") " pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.606923 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.606900 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:53.641456 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.641432 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:53.716626 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.716600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:53.778375 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.778347 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk"] Jul 21 17:06:53.781350 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:06:53.781327 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa65b591_bb96_4c3b_be95_cfd71f969bee.slice/crio-c8b816f4c611819e1e45c101981f93f2fadb5d8489cff5036b288843a18f0f49 WatchSource:0}: Error finding container c8b816f4c611819e1e45c101981f93f2fadb5d8489cff5036b288843a18f0f49: Status 404 returned error can't find the container with id c8b816f4c611819e1e45c101981f93f2fadb5d8489cff5036b288843a18f0f49 Jul 21 17:06:53.826955 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.826924 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:53.927666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:53.927581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:54.039288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:54.039260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:54.137573 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:54.137547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:54.523633 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:54.523595 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" event={"ID":"fa65b591-bb96-4c3b-be95-cfd71f969bee","Type":"ContainerStarted","Data":"c8b816f4c611819e1e45c101981f93f2fadb5d8489cff5036b288843a18f0f49"} Jul 21 17:06:55.806118 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:55.806071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:55.908195 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:55.908161 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:06:57.537528 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:57.537499 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" event={"ID":"fa65b591-bb96-4c3b-be95-cfd71f969bee","Type":"ContainerStarted","Data":"78d3a87511258b3169dcd1eb2c9bed09b7f13ef940da3c0f57cce2022cc6e71f"} Jul 21 17:06:58.541987 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:58.541952 2562 generic.go:356] "Generic (PLEG): container finished" podID="fa65b591-bb96-4c3b-be95-cfd71f969bee" containerID="78d3a87511258b3169dcd1eb2c9bed09b7f13ef940da3c0f57cce2022cc6e71f" exitCode=0 Jul 21 17:06:58.542373 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:58.542020 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" event={"ID":"fa65b591-bb96-4c3b-be95-cfd71f969bee","Type":"ContainerDied","Data":"78d3a87511258b3169dcd1eb2c9bed09b7f13ef940da3c0f57cce2022cc6e71f"} Jul 21 17:06:59.547916 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:59.547876 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" event={"ID":"fa65b591-bb96-4c3b-be95-cfd71f969bee","Type":"ContainerStarted","Data":"2ea558365f2e4b7b7eb0b19c4f415519f0a0179fe8e305140dae81c44de3a770"} Jul 21 17:06:59.548302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:59.547972 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:06:59.574202 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:06:59.574146 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" podStartSLOduration=2.058871927 podStartE2EDuration="6.574125958s" totalImagesPullingTime="3.476250602s" totalInitContainerRuntime="1.039003429s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:06:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:06:59.572968807 +0000 UTC m=+287.687115792" watchObservedRunningTime="2026-07-21 17:06:59.574125958 +0000 UTC m=+287.688272950" Jul 21 17:07:04.415907 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:04.415874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:04.517122 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:04.517072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:06.985309 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:06.985279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:07.411577 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:07.411484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:07.514506 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:07.514479 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:10.553816 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:10.553787 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk" Jul 21 17:07:12.219072 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:12.219031 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 17:07:12.390380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:12.390235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:12.555483 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:12.555401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:13.168056 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.167884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:13.287889 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.287856 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:13.388582 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.388548 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:13.530075 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.530039 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:13.815481 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.815402 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-npvmb"] Jul 21 17:07:13.818550 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.818518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:13.818694 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.818647 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:13.820755 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.820735 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 17:07:13.820978 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.820963 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-f8j29\"" Jul 21 17:07:13.832512 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.832491 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-npvmb"] Jul 21 17:07:13.835126 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.835080 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:13.835235 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.835158 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cs59\" (UniqueName: \"kubernetes.io/projected/7102ce76-14d8-4e50-af46-33c80a7c8ac9-kube-api-access-9cs59\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:13.936356 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.936317 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:13.936356 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.936350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cs59\" (UniqueName: \"kubernetes.io/projected/7102ce76-14d8-4e50-af46-33c80a7c8ac9-kube-api-access-9cs59\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:13.936606 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:07:13.936460 2562 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 21 17:07:13.936606 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:07:13.936532 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs podName:7102ce76-14d8-4e50-af46-33c80a7c8ac9 nodeName:}" failed. No retries permitted until 2026-07-21 17:07:14.436509505 +0000 UTC m=+302.550656479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs") pod "model-serving-api-54d89bd5fc-npvmb" (UID: "7102ce76-14d8-4e50-af46-33c80a7c8ac9") : secret "model-serving-api-tls" not found Jul 21 17:07:13.945877 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:13.945846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cs59\" (UniqueName: \"kubernetes.io/projected/7102ce76-14d8-4e50-af46-33c80a7c8ac9-kube-api-access-9cs59\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:14.441762 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:14.441725 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:14.442167 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:07:14.441870 2562 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 21 17:07:14.442167 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:07:14.441935 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs podName:7102ce76-14d8-4e50-af46-33c80a7c8ac9 nodeName:}" failed. No retries permitted until 2026-07-21 17:07:15.44191895 +0000 UTC m=+303.556065912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs") pod "model-serving-api-54d89bd5fc-npvmb" (UID: "7102ce76-14d8-4e50-af46-33c80a7c8ac9") : secret "model-serving-api-tls" not found Jul 21 17:07:14.634608 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:14.634556 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:15.451293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:15.451257 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:15.453650 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:15.453624 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7102ce76-14d8-4e50-af46-33c80a7c8ac9-tls-certs\") pod \"model-serving-api-54d89bd5fc-npvmb\" (UID: \"7102ce76-14d8-4e50-af46-33c80a7c8ac9\") " pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:15.628491 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:15.628457 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:15.762854 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:15.762832 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-npvmb"] Jul 21 17:07:15.765014 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:07:15.764977 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7102ce76_14d8_4e50_af46_33c80a7c8ac9.slice/crio-fd0c111a76d0f35f26288b431adb8bc91187f099411465f4a67e0c0f0aa92128 WatchSource:0}: Error finding container fd0c111a76d0f35f26288b431adb8bc91187f099411465f4a67e0c0f0aa92128: Status 404 returned error can't find the container with id fd0c111a76d0f35f26288b431adb8bc91187f099411465f4a67e0c0f0aa92128 Jul 21 17:07:15.767037 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:15.767019 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 17:07:15.978451 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:15.978411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:16.213269 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.213235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:16.292143 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.292043 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp"] Jul 21 17:07:16.295042 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.295017 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.297521 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.297498 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9g22j\"" Jul 21 17:07:16.298293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.298271 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 17:07:16.298417 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.298273 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 17:07:16.298417 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.298301 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 17:07:16.298596 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.298577 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 17:07:16.313044 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.313024 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp"] Jul 21 17:07:16.361705 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.361678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d486d602-ccaf-4538-bcf1-8b0a93a35fde-tls-certs\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.361845 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.361712 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xgvz\" (UniqueName: \"kubernetes.io/projected/d486d602-ccaf-4538-bcf1-8b0a93a35fde-kube-api-access-7xgvz\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.361845 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.361748 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d486d602-ccaf-4538-bcf1-8b0a93a35fde-tmp\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.462767 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.462740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d486d602-ccaf-4538-bcf1-8b0a93a35fde-tmp\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.463159 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.462806 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d486d602-ccaf-4538-bcf1-8b0a93a35fde-tls-certs\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.463159 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.462839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgvz\" (UniqueName: \"kubernetes.io/projected/d486d602-ccaf-4538-bcf1-8b0a93a35fde-kube-api-access-7xgvz\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.465181 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.465151 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d486d602-ccaf-4538-bcf1-8b0a93a35fde-tmp\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.465350 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.465330 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d486d602-ccaf-4538-bcf1-8b0a93a35fde-tls-certs\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.476168 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.476138 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgvz\" (UniqueName: \"kubernetes.io/projected/d486d602-ccaf-4538-bcf1-8b0a93a35fde-kube-api-access-7xgvz\") pod \"kube-auth-proxy-66868cbd5-s6nbp\" (UID: \"d486d602-ccaf-4538-bcf1-8b0a93a35fde\") " pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.605629 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.605556 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" Jul 21 17:07:16.618610 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.618575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" event={"ID":"7102ce76-14d8-4e50-af46-33c80a7c8ac9","Type":"ContainerStarted","Data":"fd0c111a76d0f35f26288b431adb8bc91187f099411465f4a67e0c0f0aa92128"} Jul 21 17:07:16.770009 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:16.769976 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp"] Jul 21 17:07:16.774044 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:07:16.774016 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd486d602_ccaf_4538_bcf1_8b0a93a35fde.slice/crio-42e2a14ca255fdcbe5f4707c5aece7afa707eb64cdff943e7f7aaf6ecf9bf02e WatchSource:0}: Error finding container 42e2a14ca255fdcbe5f4707c5aece7afa707eb64cdff943e7f7aaf6ecf9bf02e: Status 404 returned error can't find the container with id 42e2a14ca255fdcbe5f4707c5aece7afa707eb64cdff943e7f7aaf6ecf9bf02e Jul 21 17:07:17.624964 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:17.624914 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" event={"ID":"d486d602-ccaf-4538-bcf1-8b0a93a35fde","Type":"ContainerStarted","Data":"42e2a14ca255fdcbe5f4707c5aece7afa707eb64cdff943e7f7aaf6ecf9bf02e"} Jul 21 17:07:17.640899 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:17.640867 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:17.738010 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:17.737975 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:18.630784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:18.630752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" event={"ID":"7102ce76-14d8-4e50-af46-33c80a7c8ac9","Type":"ContainerStarted","Data":"755195f87d72d9eaf9be4246523728912c3cd67102dc16d5e211a8a20da3ca74"} Jul 21 17:07:18.631158 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:18.630884 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:18.652323 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:18.652280 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" podStartSLOduration=3.376288867 podStartE2EDuration="5.652266722s" totalImagesPullingTime="2.275977855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:07:18.651605984 +0000 UTC m=+306.765752969" watchObservedRunningTime="2026-07-21 17:07:18.652266722 +0000 UTC m=+306.766413707" Jul 21 17:07:19.043543 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:19.043517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:20.641321 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.641290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" event={"ID":"d486d602-ccaf-4538-bcf1-8b0a93a35fde","Type":"ContainerStarted","Data":"f41c9ab802bd788a1ee1c851959e0a903bdf919e897a29b0a31ab638f6ba135a"} Jul 21 17:07:20.662748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.662703 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66868cbd5-s6nbp" podStartSLOduration=1.483810759 podStartE2EDuration="4.662687684s" totalImagesPullingTime="3.178876925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:07:20.66076122 +0000 UTC m=+308.774908208" watchObservedRunningTime="2026-07-21 17:07:20.662687684 +0000 UTC m=+308.776834700" Jul 21 17:07:20.700603 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.700573 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7"] Jul 21 17:07:20.702781 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.702766 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.705862 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.705838 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vlbxl\"" Jul 21 17:07:20.706084 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.706050 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 17:07:20.706360 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.706346 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 17:07:20.706458 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.706356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 17:07:20.706513 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.706493 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 17:07:20.706967 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.706949 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 17:07:20.715327 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.715308 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7"] Jul 21 17:07:20.797170 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.797146 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d40707d6-2ef9-4048-9449-4503a16d2c4b-metrics-cert\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.797280 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.797199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d40707d6-2ef9-4048-9449-4503a16d2c4b-manager-config\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.797331 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.797276 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxst\" (UniqueName: \"kubernetes.io/projected/d40707d6-2ef9-4048-9449-4503a16d2c4b-kube-api-access-tgxst\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.797331 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.797309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d40707d6-2ef9-4048-9449-4503a16d2c4b-cert\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.898426 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.898357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxst\" (UniqueName: \"kubernetes.io/projected/d40707d6-2ef9-4048-9449-4503a16d2c4b-kube-api-access-tgxst\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.898426 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.898387 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d40707d6-2ef9-4048-9449-4503a16d2c4b-cert\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.898426 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.898413 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d40707d6-2ef9-4048-9449-4503a16d2c4b-metrics-cert\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.898666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.898451 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d40707d6-2ef9-4048-9449-4503a16d2c4b-manager-config\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.898998 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.898978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d40707d6-2ef9-4048-9449-4503a16d2c4b-manager-config\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.900814 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.900790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d40707d6-2ef9-4048-9449-4503a16d2c4b-metrics-cert\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.900894 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.900838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d40707d6-2ef9-4048-9449-4503a16d2c4b-cert\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:20.909728 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:20.909708 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxst\" (UniqueName: \"kubernetes.io/projected/d40707d6-2ef9-4048-9449-4503a16d2c4b-kube-api-access-tgxst\") pod \"lws-controller-manager-5f64fdf99-2ckn7\" (UID: \"d40707d6-2ef9-4048-9449-4503a16d2c4b\") " pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:21.011888 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:21.011862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:21.144367 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:21.144344 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7"] Jul 21 17:07:21.146760 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:07:21.146737 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40707d6_2ef9_4048_9449_4503a16d2c4b.slice/crio-9f4d2df81e4c75846dff5d7f29edc54e9c10fa1eee3e911ef3506759520c62c3 WatchSource:0}: Error finding container 9f4d2df81e4c75846dff5d7f29edc54e9c10fa1eee3e911ef3506759520c62c3: Status 404 returned error can't find the container with id 9f4d2df81e4c75846dff5d7f29edc54e9c10fa1eee3e911ef3506759520c62c3 Jul 21 17:07:21.646723 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:21.646681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" event={"ID":"d40707d6-2ef9-4048-9449-4503a16d2c4b","Type":"ContainerStarted","Data":"9f4d2df81e4c75846dff5d7f29edc54e9c10fa1eee3e911ef3506759520c62c3"} Jul 21 17:07:21.663872 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:21.663842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:21.846886 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:21.846860 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:22.823165 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:22.823120 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:23.310918 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:23.310880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:23.663239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:23.663158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:24.661056 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:24.661019 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" event={"ID":"d40707d6-2ef9-4048-9449-4503a16d2c4b","Type":"ContainerStarted","Data":"b0d137f3ce84f31791a76c4ba29a0893d393fbb5bdf7fb283ebf2ec7e3a5b41f"} Jul 21 17:07:24.661448 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:24.661078 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:24.698878 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:24.698824 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" podStartSLOduration=1.964721704 podStartE2EDuration="4.698805059s" totalImagesPullingTime="2.734083355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:07:24.697096746 +0000 UTC m=+312.811243730" watchObservedRunningTime="2026-07-21 17:07:24.698805059 +0000 UTC m=+312.812952044" Jul 21 17:07:24.929157 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:24.929019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:25.019698 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.019670 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc"] Jul 21 17:07:25.022144 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.022127 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.024488 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.024471 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 21 17:07:25.024611 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.024568 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-c4v8j\"" Jul 21 17:07:25.042698 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.042674 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc"] Jul 21 17:07:25.052701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.052680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:25.136612 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.136578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8shm5\" (UniqueName: \"kubernetes.io/projected/79fcc563-9de5-404b-8bc4-f8763ebdf35f-kube-api-access-8shm5\") pod \"llmisvc-controller-manager-777c6bd7d5-8vldc\" (UID: \"79fcc563-9de5-404b-8bc4-f8763ebdf35f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.136786 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.136660 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79fcc563-9de5-404b-8bc4-f8763ebdf35f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8vldc\" (UID: \"79fcc563-9de5-404b-8bc4-f8763ebdf35f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.156050 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.156021 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:25.237452 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.237418 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8shm5\" (UniqueName: \"kubernetes.io/projected/79fcc563-9de5-404b-8bc4-f8763ebdf35f-kube-api-access-8shm5\") pod \"llmisvc-controller-manager-777c6bd7d5-8vldc\" (UID: \"79fcc563-9de5-404b-8bc4-f8763ebdf35f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.237606 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.237499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79fcc563-9de5-404b-8bc4-f8763ebdf35f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8vldc\" (UID: \"79fcc563-9de5-404b-8bc4-f8763ebdf35f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.239863 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.239843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79fcc563-9de5-404b-8bc4-f8763ebdf35f-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-8vldc\" (UID: \"79fcc563-9de5-404b-8bc4-f8763ebdf35f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.252288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.252258 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8shm5\" (UniqueName: \"kubernetes.io/projected/79fcc563-9de5-404b-8bc4-f8763ebdf35f-kube-api-access-8shm5\") pod \"llmisvc-controller-manager-777c6bd7d5-8vldc\" (UID: \"79fcc563-9de5-404b-8bc4-f8763ebdf35f\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.258597 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.258566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:25.331713 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.331675 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:25.375521 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.375474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:25.510187 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.510118 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc"] Jul 21 17:07:25.530125 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:07:25.530078 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod79fcc563_9de5_404b_8bc4_f8763ebdf35f.slice/crio-81752aa1a68bdff7c778f724de9bee643078643d18d72887b99807df9aa584a6 WatchSource:0}: Error finding container 81752aa1a68bdff7c778f724de9bee643078643d18d72887b99807df9aa584a6: Status 404 returned error can't find the container with id 81752aa1a68bdff7c778f724de9bee643078643d18d72887b99807df9aa584a6 Jul 21 17:07:25.666686 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:25.666655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" event={"ID":"79fcc563-9de5-404b-8bc4-f8763ebdf35f","Type":"ContainerStarted","Data":"81752aa1a68bdff7c778f724de9bee643078643d18d72887b99807df9aa584a6"} Jul 21 17:07:29.640587 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:29.640559 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-npvmb" Jul 21 17:07:29.690681 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:29.690651 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" event={"ID":"79fcc563-9de5-404b-8bc4-f8763ebdf35f","Type":"ContainerStarted","Data":"4722409167dfc3e00fe99d2cd9b64a1dfffafab578cb817617665dc82c0617aa"} Jul 21 17:07:29.690846 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:29.690759 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:07:29.954795 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:29.954723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:30.057834 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:30.057767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:07:35.669206 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:35.669174 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f64fdf99-2ckn7" Jul 21 17:07:35.690243 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:07:35.690195 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" podStartSLOduration=8.437830181 podStartE2EDuration="11.690179813s" totalImagesPullingTime="3.252349632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:07:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:07:29.716384184 +0000 UTC m=+317.830531165" watchObservedRunningTime="2026-07-21 17:07:35.690179813 +0000 UTC m=+323.804326797" Jul 21 17:08:00.697364 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:00.697337 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-8vldc" Jul 21 17:08:13.942611 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:13.942578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:14.100523 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:14.100492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:14.205961 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:14.205884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:35.414919 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:35.414885 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:35.516515 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:35.516484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:37.466279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:37.466246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:39.393593 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:39.393558 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:39.542139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:39.542090 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:39.604923 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:39.604898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:40.150963 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:40.150934 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:40.219561 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:40.219532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:43.677732 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:43.677707 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:46.872359 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:46.872328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:47.668086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:47.668051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:50.615142 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:50.615085 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:50.722437 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:50.722405 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:50.820379 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:50.820345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:50.919933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:50.919855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:53.591393 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:53.591361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:55.118648 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:55.118617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:56.275847 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:56.275813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:58.334710 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:58.334681 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:08:59.927122 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:08:59.927075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:00.031409 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:00.031377 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:03.873663 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:03.873630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:03.970091 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:03.970061 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:04.281993 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:04.281954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:04.382317 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:04.382282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:05.460601 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.460526 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw"] Jul 21 17:09:05.463880 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.463862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.466694 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.466676 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 17:09:05.466777 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.466679 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 17:09:05.467002 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.466988 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kmzmg\"" Jul 21 17:09:05.467551 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.467537 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 17:09:05.467670 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.467659 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 17:09:05.473675 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.473655 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw"] Jul 21 17:09:05.477937 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.477918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:05.564518 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.564487 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz249\" (UniqueName: \"kubernetes.io/projected/623f7981-b0d0-4897-afd1-1d822b56e8ac-kube-api-access-lz249\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.564633 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.564528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/623f7981-b0d0-4897-afd1-1d822b56e8ac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.564633 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.564564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.665981 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.665953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz249\" (UniqueName: \"kubernetes.io/projected/623f7981-b0d0-4897-afd1-1d822b56e8ac-kube-api-access-lz249\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.666097 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.665985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/623f7981-b0d0-4897-afd1-1d822b56e8ac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.666097 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.666019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.666097 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:09:05.666097 2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 21 17:09:05.666217 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:09:05.666162 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert podName:623f7981-b0d0-4897-afd1-1d822b56e8ac nodeName:}" failed. No retries permitted until 2026-07-21 17:09:06.166149263 +0000 UTC m=+414.280296225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-bsdhw" (UID: "623f7981-b0d0-4897-afd1-1d822b56e8ac") : secret "plugin-serving-cert" not found Jul 21 17:09:05.666616 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.666597 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/623f7981-b0d0-4897-afd1-1d822b56e8ac-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:05.677858 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:05.677828 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz249\" (UniqueName: \"kubernetes.io/projected/623f7981-b0d0-4897-afd1-1d822b56e8ac-kube-api-access-lz249\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:06.170691 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:06.170658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:06.170859 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:09:06.170766 2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 21 17:09:06.170859 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:09:06.170820 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert podName:623f7981-b0d0-4897-afd1-1d822b56e8ac nodeName:}" failed. No retries permitted until 2026-07-21 17:09:07.170807663 +0000 UTC m=+415.284954626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-bsdhw" (UID: "623f7981-b0d0-4897-afd1-1d822b56e8ac") : secret "plugin-serving-cert" not found Jul 21 17:09:06.226726 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:06.226696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:07.179366 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:07.179336 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:07.181664 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:07.181644 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/623f7981-b0d0-4897-afd1-1d822b56e8ac-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bsdhw\" (UID: \"623f7981-b0d0-4897-afd1-1d822b56e8ac\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:07.272747 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:07.272710 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" Jul 21 17:09:07.403653 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:07.403622 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw"] Jul 21 17:09:07.406709 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:09:07.406685 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod623f7981_b0d0_4897_afd1_1d822b56e8ac.slice/crio-e90720c85d44f12a4ba9dcf07d066a34be614e9646aba7a3b265313591812f9e WatchSource:0}: Error finding container e90720c85d44f12a4ba9dcf07d066a34be614e9646aba7a3b265313591812f9e: Status 404 returned error can't find the container with id e90720c85d44f12a4ba9dcf07d066a34be614e9646aba7a3b265313591812f9e Jul 21 17:09:08.098022 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:08.097989 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" event={"ID":"623f7981-b0d0-4897-afd1-1d822b56e8ac","Type":"ContainerStarted","Data":"e90720c85d44f12a4ba9dcf07d066a34be614e9646aba7a3b265313591812f9e"} Jul 21 17:09:08.413344 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:08.413233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:08.510980 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:08.510931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:14.383087 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:14.383040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:17.189598 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:17.189532 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:17.287622 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:17.287589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:22.747885 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:22.747852 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:22.847017 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:22.846957 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:28.057815 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:28.057752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:28.165312 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:28.165271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:28.598670 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:28.598640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:28.704013 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:28.703982 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:31.217048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:31.217011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" event={"ID":"623f7981-b0d0-4897-afd1-1d822b56e8ac","Type":"ContainerStarted","Data":"c160fce2c7f7e1bfe98a278fa371c7abfd91976232bf7d86c853a7c111fb70dd"} Jul 21 17:09:32.400185 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:32.400153 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:32.499087 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:32.499053 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:38.905271 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:38.905167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:39.007351 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:39.007319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:47.410554 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:47.410518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:47.510876 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:47.510832 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:48.756783 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:48.756748 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:09:49.033984 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:09:49.033890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:04.368415 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.368380 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:04.484429 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.484395 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:04.587585 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.587552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:04.823547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.823483 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bsdhw" podStartSLOduration=36.628212152 podStartE2EDuration="59.823461314s" totalImagesPullingTime="23.195249162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:09:31.242645298 +0000 UTC m=+439.356792282" watchObservedRunningTime="2026-07-21 17:10:04.823461314 +0000 UTC m=+472.937608298" Jul 21 17:10:04.824284 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.824261 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:10:04.827303 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.827285 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:04.829994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.829976 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 17:10:04.837235 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.837216 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:10:04.874051 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.874020 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:10:04.894005 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.893979 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a3f56fa-75cd-47cc-afd6-99e9aae217c9-config-file\") pod \"limitador-limitador-54d6d988d5-8q8h7\" (UID: \"0a3f56fa-75cd-47cc-afd6-99e9aae217c9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:04.894150 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.894018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9xds\" (UniqueName: \"kubernetes.io/projected/0a3f56fa-75cd-47cc-afd6-99e9aae217c9-kube-api-access-c9xds\") pod \"limitador-limitador-54d6d988d5-8q8h7\" (UID: \"0a3f56fa-75cd-47cc-afd6-99e9aae217c9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:04.995455 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.995433 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a3f56fa-75cd-47cc-afd6-99e9aae217c9-config-file\") pod \"limitador-limitador-54d6d988d5-8q8h7\" (UID: \"0a3f56fa-75cd-47cc-afd6-99e9aae217c9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:04.995539 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.995472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9xds\" (UniqueName: \"kubernetes.io/projected/0a3f56fa-75cd-47cc-afd6-99e9aae217c9-kube-api-access-c9xds\") pod \"limitador-limitador-54d6d988d5-8q8h7\" (UID: \"0a3f56fa-75cd-47cc-afd6-99e9aae217c9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:04.996043 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:04.996026 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0a3f56fa-75cd-47cc-afd6-99e9aae217c9-config-file\") pod \"limitador-limitador-54d6d988d5-8q8h7\" (UID: \"0a3f56fa-75cd-47cc-afd6-99e9aae217c9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:05.004789 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.004762 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9xds\" (UniqueName: \"kubernetes.io/projected/0a3f56fa-75cd-47cc-afd6-99e9aae217c9-kube-api-access-c9xds\") pod \"limitador-limitador-54d6d988d5-8q8h7\" (UID: \"0a3f56fa-75cd-47cc-afd6-99e9aae217c9\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:05.137603 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.137518 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:05.267207 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.267179 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-kc8sv"] Jul 21 17:10:05.269172 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:10:05.269146 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a3f56fa_75cd_47cc_afd6_99e9aae217c9.slice/crio-bd5b07124a1172cb8808cbf1cd831bd0b896d85ec43889719d6903c4b41b7494 WatchSource:0}: Error finding container bd5b07124a1172cb8808cbf1cd831bd0b896d85ec43889719d6903c4b41b7494: Status 404 returned error can't find the container with id bd5b07124a1172cb8808cbf1cd831bd0b896d85ec43889719d6903c4b41b7494 Jul 21 17:10:05.270840 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.270789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:10:05.270922 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.270905 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kc8sv" Jul 21 17:10:05.273418 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.273392 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7qs8z\"" Jul 21 17:10:05.276827 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.276807 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kc8sv"] Jul 21 17:10:05.298298 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.298271 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmwd\" (UniqueName: \"kubernetes.io/projected/0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac-kube-api-access-ctmwd\") pod \"authorino-58cd955675-kc8sv\" (UID: \"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac\") " pod="kuadrant-system/authorino-58cd955675-kc8sv" Jul 21 17:10:05.304732 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.304711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:05.363429 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.363400 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" event={"ID":"0a3f56fa-75cd-47cc-afd6-99e9aae217c9","Type":"ContainerStarted","Data":"bd5b07124a1172cb8808cbf1cd831bd0b896d85ec43889719d6903c4b41b7494"} Jul 21 17:10:05.375508 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.375481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:05.399559 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.399302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmwd\" (UniqueName: \"kubernetes.io/projected/0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac-kube-api-access-ctmwd\") pod \"authorino-58cd955675-kc8sv\" (UID: \"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac\") " pod="kuadrant-system/authorino-58cd955675-kc8sv" Jul 21 17:10:05.408366 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.408344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmwd\" (UniqueName: \"kubernetes.io/projected/0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac-kube-api-access-ctmwd\") pod \"authorino-58cd955675-kc8sv\" (UID: \"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac\") " pod="kuadrant-system/authorino-58cd955675-kc8sv" Jul 21 17:10:05.585000 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.584963 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kc8sv" Jul 21 17:10:05.717560 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:05.717530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kc8sv"] Jul 21 17:10:05.720804 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:10:05.720770 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b63c3d8_06b2_4e92_8f5a_4bf1042ca7ac.slice/crio-555569236c92602648f0accc1fc232b4ae267ded48bde786436fc3e5869980db WatchSource:0}: Error finding container 555569236c92602648f0accc1fc232b4ae267ded48bde786436fc3e5869980db: Status 404 returned error can't find the container with id 555569236c92602648f0accc1fc232b4ae267ded48bde786436fc3e5869980db Jul 21 17:10:06.369860 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:06.369825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kc8sv" event={"ID":"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac","Type":"ContainerStarted","Data":"555569236c92602648f0accc1fc232b4ae267ded48bde786436fc3e5869980db"} Jul 21 17:10:09.856654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:09.856568 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kc8sv"] Jul 21 17:10:10.393255 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:10.393216 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" event={"ID":"0a3f56fa-75cd-47cc-afd6-99e9aae217c9","Type":"ContainerStarted","Data":"68e734cbe1ad8874a2bd8d93bd206560dd4aa839a0e924ca4f4a5c148fccee73"} Jul 21 17:10:10.393454 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:10.393334 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:10.394595 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:10.394571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kc8sv" event={"ID":"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac","Type":"ContainerStarted","Data":"fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e"} Jul 21 17:10:10.411119 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:10.411062 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" podStartSLOduration=2.097185555 podStartE2EDuration="6.411043337s" totalImagesPullingTime="4.313857782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:10:10.410133622 +0000 UTC m=+478.524280609" watchObservedRunningTime="2026-07-21 17:10:10.411043337 +0000 UTC m=+478.525190324" Jul 21 17:10:10.424555 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:10.424511 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-kc8sv" podStartSLOduration=1.5537538770000001 podStartE2EDuration="5.424494303s" totalImagesPullingTime="3.870740426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:10:10.42426558 +0000 UTC m=+478.538412579" watchObservedRunningTime="2026-07-21 17:10:10.424494303 +0000 UTC m=+478.538641280" Jul 21 17:10:11.399446 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:11.399408 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-kc8sv" podUID="0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac" containerName="authorino" containerID="cri-o://fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e" gracePeriod=30 Jul 21 17:10:11.649211 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:11.649188 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kc8sv" Jul 21 17:10:11.759453 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:11.759423 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctmwd\" (UniqueName: \"kubernetes.io/projected/0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac-kube-api-access-ctmwd\") pod \"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac\" (UID: \"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac\") " Jul 21 17:10:11.761566 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:11.761528 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac-kube-api-access-ctmwd" (OuterVolumeSpecName: "kube-api-access-ctmwd") pod "0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac" (UID: "0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac"). InnerVolumeSpecName "kube-api-access-ctmwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:10:11.860302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:11.860254 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmwd\" (UniqueName: \"kubernetes.io/projected/0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac-kube-api-access-ctmwd\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:10:12.404340 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.404310 2562 generic.go:356] "Generic (PLEG): container finished" podID="0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac" containerID="fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e" exitCode=0 Jul 21 17:10:12.404738 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.404354 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kc8sv" Jul 21 17:10:12.404738 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.404369 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kc8sv" event={"ID":"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac","Type":"ContainerDied","Data":"fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e"} Jul 21 17:10:12.404738 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.404404 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kc8sv" event={"ID":"0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac","Type":"ContainerDied","Data":"555569236c92602648f0accc1fc232b4ae267ded48bde786436fc3e5869980db"} Jul 21 17:10:12.404738 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.404423 2562 scope.go:117] "RemoveContainer" containerID="fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e" Jul 21 17:10:12.414682 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.414662 2562 scope.go:117] "RemoveContainer" containerID="fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e" Jul 21 17:10:12.414980 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:10:12.414959 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e\": container with ID starting with fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e not found: ID does not exist" containerID="fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e" Jul 21 17:10:12.415026 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.414991 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e"} err="failed to get container status \"fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e\": rpc error: code = NotFound desc = could not find container \"fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e\": container with ID starting with fb2b43e3f04f0a1bb061f222badddcef1fe672cf3e95ea63efee8d980739ca7e not found: ID does not exist" Jul 21 17:10:12.423813 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.423784 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kc8sv"] Jul 21 17:10:12.431748 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:12.429639 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-kc8sv"] Jul 21 17:10:14.377787 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:14.377753 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac" path="/var/lib/kubelet/pods/0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac/volumes" Jul 21 17:10:21.400808 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:21.400778 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8q8h7" Jul 21 17:10:27.068555 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:27.068519 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:27.170261 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:27.170222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:27.566430 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:27.566391 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:39.985262 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:39.985228 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:40.206728 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.206695 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-z8mvn"] Jul 21 17:10:40.207082 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.207070 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac" containerName="authorino" Jul 21 17:10:40.207151 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.207084 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac" containerName="authorino" Jul 21 17:10:40.207188 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.207155 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b63c3d8-06b2-4e92-8f5a-4bf1042ca7ac" containerName="authorino" Jul 21 17:10:40.213070 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.213042 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" Jul 21 17:10:40.213393 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.213372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t792\" (UniqueName: \"kubernetes.io/projected/6e327ef3-53a6-44be-94e8-81dac7dbe73c-kube-api-access-4t792\") pod \"authorino-5d4b67c446-z8mvn\" (UID: \"6e327ef3-53a6-44be-94e8-81dac7dbe73c\") " pod="kuadrant-system/authorino-5d4b67c446-z8mvn" Jul 21 17:10:40.215457 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.215436 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7qs8z\"" Jul 21 17:10:40.215553 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.215455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-z8mvn"] Jul 21 17:10:40.223931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.223907 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:40.314423 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.314355 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t792\" (UniqueName: \"kubernetes.io/projected/6e327ef3-53a6-44be-94e8-81dac7dbe73c-kube-api-access-4t792\") pod \"authorino-5d4b67c446-z8mvn\" (UID: \"6e327ef3-53a6-44be-94e8-81dac7dbe73c\") " pod="kuadrant-system/authorino-5d4b67c446-z8mvn" Jul 21 17:10:40.323856 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.323831 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t792\" (UniqueName: \"kubernetes.io/projected/6e327ef3-53a6-44be-94e8-81dac7dbe73c-kube-api-access-4t792\") pod \"authorino-5d4b67c446-z8mvn\" (UID: \"6e327ef3-53a6-44be-94e8-81dac7dbe73c\") " pod="kuadrant-system/authorino-5d4b67c446-z8mvn" Jul 21 17:10:40.430239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.430214 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-z8mvn"] Jul 21 17:10:40.430388 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.430377 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" Jul 21 17:10:40.459460 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.459433 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6754694d8d-plfn8"] Jul 21 17:10:40.464833 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.464808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754694d8d-plfn8" Jul 21 17:10:40.472636 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.472120 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6754694d8d-plfn8"] Jul 21 17:10:40.516878 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.516851 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhkv\" (UniqueName: \"kubernetes.io/projected/11f38493-22df-4c2d-afb3-f45643c95098-kube-api-access-pjhkv\") pod \"authorino-6754694d8d-plfn8\" (UID: \"11f38493-22df-4c2d-afb3-f45643c95098\") " pod="kuadrant-system/authorino-6754694d8d-plfn8" Jul 21 17:10:40.543884 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.543849 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6754694d8d-plfn8"] Jul 21 17:10:40.544174 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:10:40.544151 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-pjhkv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6754694d8d-plfn8" podUID="11f38493-22df-4c2d-afb3-f45643c95098" Jul 21 17:10:40.566187 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.566166 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-z8mvn"] Jul 21 17:10:40.568747 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:10:40.568724 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e327ef3_53a6_44be_94e8_81dac7dbe73c.slice/crio-0b6f4bf4161d026767ca8f7b30f100346fd809e6b4aee6aafa7fc037187bb69e WatchSource:0}: Error finding container 0b6f4bf4161d026767ca8f7b30f100346fd809e6b4aee6aafa7fc037187bb69e: Status 404 returned error can't find the container with id 0b6f4bf4161d026767ca8f7b30f100346fd809e6b4aee6aafa7fc037187bb69e Jul 21 17:10:40.617687 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.617665 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhkv\" (UniqueName: \"kubernetes.io/projected/11f38493-22df-4c2d-afb3-f45643c95098-kube-api-access-pjhkv\") pod \"authorino-6754694d8d-plfn8\" (UID: \"11f38493-22df-4c2d-afb3-f45643c95098\") " pod="kuadrant-system/authorino-6754694d8d-plfn8" Jul 21 17:10:40.626781 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:40.626757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhkv\" (UniqueName: \"kubernetes.io/projected/11f38493-22df-4c2d-afb3-f45643c95098-kube-api-access-pjhkv\") pod \"authorino-6754694d8d-plfn8\" (UID: \"11f38493-22df-4c2d-afb3-f45643c95098\") " pod="kuadrant-system/authorino-6754694d8d-plfn8" Jul 21 17:10:41.530620 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.530591 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754694d8d-plfn8" Jul 21 17:10:41.531060 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.530646 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" podUID="6e327ef3-53a6-44be-94e8-81dac7dbe73c" containerName="authorino" containerID="cri-o://9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114" gracePeriod=30 Jul 21 17:10:41.531060 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.530641 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" event={"ID":"6e327ef3-53a6-44be-94e8-81dac7dbe73c","Type":"ContainerStarted","Data":"9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114"} Jul 21 17:10:41.531060 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.530775 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" event={"ID":"6e327ef3-53a6-44be-94e8-81dac7dbe73c","Type":"ContainerStarted","Data":"0b6f4bf4161d026767ca8f7b30f100346fd809e6b4aee6aafa7fc037187bb69e"} Jul 21 17:10:41.554046 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.554026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754694d8d-plfn8" Jul 21 17:10:41.561781 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.561743 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" podStartSLOduration=1.083383354 podStartE2EDuration="1.561729807s" totalImagesPullingTime="478.346453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:10:41.560280698 +0000 UTC m=+509.674427693" watchObservedRunningTime="2026-07-21 17:10:41.561729807 +0000 UTC m=+509.675876825" Jul 21 17:10:41.626638 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.626610 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjhkv\" (UniqueName: \"kubernetes.io/projected/11f38493-22df-4c2d-afb3-f45643c95098-kube-api-access-pjhkv\") pod \"11f38493-22df-4c2d-afb3-f45643c95098\" (UID: \"11f38493-22df-4c2d-afb3-f45643c95098\") " Jul 21 17:10:41.628691 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.628653 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f38493-22df-4c2d-afb3-f45643c95098-kube-api-access-pjhkv" (OuterVolumeSpecName: "kube-api-access-pjhkv") pod "11f38493-22df-4c2d-afb3-f45643c95098" (UID: "11f38493-22df-4c2d-afb3-f45643c95098"). InnerVolumeSpecName "kube-api-access-pjhkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:10:41.727830 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.727801 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhkv\" (UniqueName: \"kubernetes.io/projected/11f38493-22df-4c2d-afb3-f45643c95098-kube-api-access-pjhkv\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:10:41.782831 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.782776 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" Jul 21 17:10:41.828887 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.828849 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t792\" (UniqueName: \"kubernetes.io/projected/6e327ef3-53a6-44be-94e8-81dac7dbe73c-kube-api-access-4t792\") pod \"6e327ef3-53a6-44be-94e8-81dac7dbe73c\" (UID: \"6e327ef3-53a6-44be-94e8-81dac7dbe73c\") " Jul 21 17:10:41.830932 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.830904 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e327ef3-53a6-44be-94e8-81dac7dbe73c-kube-api-access-4t792" (OuterVolumeSpecName: "kube-api-access-4t792") pod "6e327ef3-53a6-44be-94e8-81dac7dbe73c" (UID: "6e327ef3-53a6-44be-94e8-81dac7dbe73c"). InnerVolumeSpecName "kube-api-access-4t792". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:10:41.930533 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:41.930496 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t792\" (UniqueName: \"kubernetes.io/projected/6e327ef3-53a6-44be-94e8-81dac7dbe73c-kube-api-access-4t792\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:10:42.537325 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.537295 2562 generic.go:356] "Generic (PLEG): container finished" podID="6e327ef3-53a6-44be-94e8-81dac7dbe73c" containerID="9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114" exitCode=0 Jul 21 17:10:42.537733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.537337 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" Jul 21 17:10:42.537733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.537376 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" event={"ID":"6e327ef3-53a6-44be-94e8-81dac7dbe73c","Type":"ContainerDied","Data":"9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114"} Jul 21 17:10:42.537733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.537415 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-z8mvn" event={"ID":"6e327ef3-53a6-44be-94e8-81dac7dbe73c","Type":"ContainerDied","Data":"0b6f4bf4161d026767ca8f7b30f100346fd809e6b4aee6aafa7fc037187bb69e"} Jul 21 17:10:42.537733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.537435 2562 scope.go:117] "RemoveContainer" containerID="9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114" Jul 21 17:10:42.537733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.537522 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754694d8d-plfn8" Jul 21 17:10:42.546252 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.546232 2562 scope.go:117] "RemoveContainer" containerID="9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114" Jul 21 17:10:42.546493 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:10:42.546475 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114\": container with ID starting with 9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114 not found: ID does not exist" containerID="9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114" Jul 21 17:10:42.546578 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.546503 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114"} err="failed to get container status \"9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114\": rpc error: code = NotFound desc = could not find container \"9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114\": container with ID starting with 9206e20b81ecf17711434d6d8a2834ab3cc1bd2d815a5d9c4f062a59d3ed9114 not found: ID does not exist" Jul 21 17:10:42.572390 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.572362 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6754694d8d-plfn8"] Jul 21 17:10:42.577526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.577505 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6754694d8d-plfn8"] Jul 21 17:10:42.588910 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.588858 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-z8mvn"] Jul 21 17:10:42.592151 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:42.592133 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-z8mvn"] Jul 21 17:10:44.377702 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:44.377665 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11f38493-22df-4c2d-afb3-f45643c95098" path="/var/lib/kubelet/pods/11f38493-22df-4c2d-afb3-f45643c95098/volumes" Jul 21 17:10:44.378211 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:44.378003 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e327ef3-53a6-44be-94e8-81dac7dbe73c" path="/var/lib/kubelet/pods/6e327ef3-53a6-44be-94e8-81dac7dbe73c/volumes" Jul 21 17:10:46.565906 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:46.565881 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:46.667907 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:46.667878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:49.535570 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:49.535537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:10:49.636737 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:10:49.636696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:20.679575 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:20.679536 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:20.776386 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:20.776353 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:21.328915 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:21.328884 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:21.685490 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:21.685420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:21.970683 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:21.970645 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:22.069323 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:22.069288 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:22.544782 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:22.541842 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:11:33.941421 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:33.941388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:34.069344 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:34.069316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:37.281189 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:37.281155 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:37.382279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:37.382245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:50.683270 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:50.683235 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:51.215666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:51.215629 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:11:57.354914 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:57.354857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:11:58.213702 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:11:58.213668 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:12:04.823006 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:04.822962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:12:06.030431 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:06.030401 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:12:20.739909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.739875 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p"] Jul 21 17:12:20.741979 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.740221 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e327ef3-53a6-44be-94e8-81dac7dbe73c" containerName="authorino" Jul 21 17:12:20.741979 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.740232 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e327ef3-53a6-44be-94e8-81dac7dbe73c" containerName="authorino" Jul 21 17:12:20.741979 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.740300 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e327ef3-53a6-44be-94e8-81dac7dbe73c" containerName="authorino" Jul 21 17:12:20.742504 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.742489 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.745894 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.745867 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k6wdl\"" Jul 21 17:12:20.745894 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.745879 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 17:12:20.746087 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.745897 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 17:12:20.746087 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.745916 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 17:12:20.754665 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.754645 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p"] Jul 21 17:12:20.758308 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.758290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.758406 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.758314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjzx8\" (UniqueName: \"kubernetes.io/projected/ca8f4c33-04a2-4365-abde-33a06247e05d-kube-api-access-cjzx8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.758406 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.758341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.758406 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.758357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.758601 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.758435 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.758601 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.758471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8f4c33-04a2-4365-abde-33a06247e05d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.788448 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.788426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:12:20.859738 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.859708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.859904 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.859741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjzx8\" (UniqueName: \"kubernetes.io/projected/ca8f4c33-04a2-4365-abde-33a06247e05d-kube-api-access-cjzx8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.859904 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.859782 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.859904 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.859798 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.859904 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.859825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.860134 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.859916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8f4c33-04a2-4365-abde-33a06247e05d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.860209 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.860191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.860266 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.860239 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.860387 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.860364 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.862216 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.862194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca8f4c33-04a2-4365-abde-33a06247e05d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.862333 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.862319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca8f4c33-04a2-4365-abde-33a06247e05d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:20.869842 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:20.869817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjzx8\" (UniqueName: \"kubernetes.io/projected/ca8f4c33-04a2-4365-abde-33a06247e05d-kube-api-access-cjzx8\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p\" (UID: \"ca8f4c33-04a2-4365-abde-33a06247e05d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:21.052968 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:21.052891 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:21.181530 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:21.181508 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p"] Jul 21 17:12:21.184026 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:12:21.184001 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8f4c33_04a2_4365_abde_33a06247e05d.slice/crio-e0f3272fc03ec0d13cfb1d47e39f2080f311d193adf7a5845a848b6810e85145 WatchSource:0}: Error finding container e0f3272fc03ec0d13cfb1d47e39f2080f311d193adf7a5845a848b6810e85145: Status 404 returned error can't find the container with id e0f3272fc03ec0d13cfb1d47e39f2080f311d193adf7a5845a848b6810e85145 Jul 21 17:12:21.185374 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:21.185355 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 17:12:21.967367 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:21.967334 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" event={"ID":"ca8f4c33-04a2-4365-abde-33a06247e05d","Type":"ContainerStarted","Data":"e0f3272fc03ec0d13cfb1d47e39f2080f311d193adf7a5845a848b6810e85145"} Jul 21 17:12:22.312005 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:22.311969 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:12:22.974012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:22.973977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" event={"ID":"ca8f4c33-04a2-4365-abde-33a06247e05d","Type":"ContainerStarted","Data":"bb0989baf2a480b9df42bb30568f5d1c8bc2ddd28f7cf7f2cc8429066798e617"} Jul 21 17:12:22.995966 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:22.995915 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" podStartSLOduration=1.985147944 podStartE2EDuration="2.995900442s" totalImagesPullingTime="1.010752498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:12:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:12:22.99485556 +0000 UTC m=+611.109002557" watchObservedRunningTime="2026-07-21 17:12:22.995900442 +0000 UTC m=+611.110047427" Jul 21 17:12:26.578220 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:26.578189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:12:29.524132 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:29.522988 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:12:31.053818 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:31.053782 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:31.054229 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:31.053846 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:31.066011 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:31.065988 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:32.024974 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:32.024947 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p" Jul 21 17:12:51.937026 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:51.936936 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z"] Jul 21 17:12:51.942713 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:51.942693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:51.945440 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:51.945422 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 17:12:51.951317 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:51.951297 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z"] Jul 21 17:12:51.979250 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:51.979217 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:12:52.037998 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.037972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.038176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.038008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5rbf\" (UniqueName: \"kubernetes.io/projected/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-kube-api-access-x5rbf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.038176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.038051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.038176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.038081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.038176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.038128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.038176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.038163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.138983 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.138956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139156 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.138992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139156 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.139031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139156 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.139091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139156 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.139152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5rbf\" (UniqueName: \"kubernetes.io/projected/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-kube-api-access-x5rbf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139355 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.139207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139552 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.139519 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139674 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.139551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.139674 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.139576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.141271 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.141251 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.141441 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.141425 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.147885 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.147862 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5rbf\" (UniqueName: \"kubernetes.io/projected/3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7-kube-api-access-x5rbf\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z\" (UID: \"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.252227 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.252192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:12:52.382678 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:52.382657 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z"] Jul 21 17:12:52.385299 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:12:52.385272 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c102820_9a1d_4ea1_b1f8_4958c4c0e3f7.slice/crio-963d036eee9c703eca8bff912501456dc8e9417634d6a076af1bf03ae0053c08 WatchSource:0}: Error finding container 963d036eee9c703eca8bff912501456dc8e9417634d6a076af1bf03ae0053c08: Status 404 returned error can't find the container with id 963d036eee9c703eca8bff912501456dc8e9417634d6a076af1bf03ae0053c08 Jul 21 17:12:53.109162 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:53.109119 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" event={"ID":"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7","Type":"ContainerStarted","Data":"bece9eb905f5949554b14281d6672b5efafbfbd185bf44f76a0864ed946da481"} Jul 21 17:12:53.109162 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:53.109168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" event={"ID":"3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7","Type":"ContainerStarted","Data":"963d036eee9c703eca8bff912501456dc8e9417634d6a076af1bf03ae0053c08"} Jul 21 17:12:53.140805 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:53.140758 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" podStartSLOduration=1.963087892 podStartE2EDuration="2.140739843s" totalImagesPullingTime="177.651951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:12:53.139437632 +0000 UTC m=+641.253584617" watchObservedRunningTime="2026-07-21 17:12:53.140739843 +0000 UTC m=+641.254886809" Jul 21 17:12:55.623012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:12:55.622977 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:13:02.252994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:02.252958 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:13:02.253502 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:02.253263 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:13:02.265659 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:02.265631 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:13:03.163943 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:03.163914 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z" Jul 21 17:13:08.607029 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:08.606987 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:08.671021 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:08.670994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:08.867064 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:08.866976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:09.009498 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:09.009472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:14.885810 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:14.885775 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:14.953060 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:14.953032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:18.187127 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:18.187079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:47.877359 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:47.877328 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-584899858f-5f22d"] Jul 21 17:13:47.879591 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:47.879577 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:47.882350 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:47.882323 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 17:13:47.882466 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:47.882350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7qs8z\"" Jul 21 17:13:47.888574 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:47.888554 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584899858f-5f22d"] Jul 21 17:13:48.036327 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.036292 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/157ed407-95c2-4154-a862-4fd938b2bf2b-tls-cert\") pod \"authorino-584899858f-5f22d\" (UID: \"157ed407-95c2-4154-a862-4fd938b2bf2b\") " pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:48.036499 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.036383 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6xz2\" (UniqueName: \"kubernetes.io/projected/157ed407-95c2-4154-a862-4fd938b2bf2b-kube-api-access-v6xz2\") pod \"authorino-584899858f-5f22d\" (UID: \"157ed407-95c2-4154-a862-4fd938b2bf2b\") " pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:48.098427 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.098394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:48.136895 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.136824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6xz2\" (UniqueName: \"kubernetes.io/projected/157ed407-95c2-4154-a862-4fd938b2bf2b-kube-api-access-v6xz2\") pod \"authorino-584899858f-5f22d\" (UID: \"157ed407-95c2-4154-a862-4fd938b2bf2b\") " pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:48.137026 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.136912 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/157ed407-95c2-4154-a862-4fd938b2bf2b-tls-cert\") pod \"authorino-584899858f-5f22d\" (UID: \"157ed407-95c2-4154-a862-4fd938b2bf2b\") " pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:48.139317 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.139297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/157ed407-95c2-4154-a862-4fd938b2bf2b-tls-cert\") pod \"authorino-584899858f-5f22d\" (UID: \"157ed407-95c2-4154-a862-4fd938b2bf2b\") " pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:48.148359 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.148334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6xz2\" (UniqueName: \"kubernetes.io/projected/157ed407-95c2-4154-a862-4fd938b2bf2b-kube-api-access-v6xz2\") pod \"authorino-584899858f-5f22d\" (UID: \"157ed407-95c2-4154-a862-4fd938b2bf2b\") " pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:48.189305 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.189281 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-584899858f-5f22d" Jul 21 17:13:48.320574 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.320545 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:48.341284 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.341259 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-584899858f-5f22d"] Jul 21 17:13:48.343806 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:13:48.343778 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod157ed407_95c2_4154_a862_4fd938b2bf2b.slice/crio-dddd8fd47083413f936e92a35bb25971efbb311488219b12b1a0bda0d11c86ff WatchSource:0}: Error finding container dddd8fd47083413f936e92a35bb25971efbb311488219b12b1a0bda0d11c86ff: Status 404 returned error can't find the container with id dddd8fd47083413f936e92a35bb25971efbb311488219b12b1a0bda0d11c86ff Jul 21 17:13:48.358495 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:48.358466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584899858f-5f22d" event={"ID":"157ed407-95c2-4154-a862-4fd938b2bf2b","Type":"ContainerStarted","Data":"dddd8fd47083413f936e92a35bb25971efbb311488219b12b1a0bda0d11c86ff"} Jul 21 17:13:49.207189 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:49.207083 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:49.321602 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:49.321568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:13:49.365711 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:49.365677 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-584899858f-5f22d" event={"ID":"157ed407-95c2-4154-a862-4fd938b2bf2b","Type":"ContainerStarted","Data":"0dadd5aaed98648789dfeae5537d8a3d26e9383d5ba66739fc0e2789cd6d106c"} Jul 21 17:13:49.395532 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:13:49.395488 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-584899858f-5f22d" podStartSLOduration=1.9966879020000001 podStartE2EDuration="2.395471321s" totalImagesPullingTime="398.783419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:13:49.395238772 +0000 UTC m=+697.509385758" watchObservedRunningTime="2026-07-21 17:13:49.395471321 +0000 UTC m=+697.509618305" Jul 21 17:14:17.551480 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:14:17.551442 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:14:17.644056 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:14:17.644008 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:14:17.747785 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:14:17.747753 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:14:26.822085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:14:26.822052 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:14:37.122071 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:14:37.122038 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:14:40.726167 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:14:40.726126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:15:54.067772 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:15:54.067667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:15:55.297085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:15:55.297049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:19.928279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:19.928249 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:16:29.009960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:29.009928 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:16:29.305239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:29.305155 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:29.404659 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:29.404627 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:29.520001 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:29.519960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:36.243031 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:36.242997 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:36.343720 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:36.343692 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:36.444137 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:36.444068 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:42.832815 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:42.832784 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:16:42.911648 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:16:42.911619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:17:01.219497 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:17:01.219458 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:17:10.321669 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:17:10.321633 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:17:20.614942 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:17:20.614907 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:17:28.216941 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:17:28.216906 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:17:37.938293 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:17:37.938257 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:17:54.126153 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:17:54.126089 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:18:00.622620 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:18:00.622584 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:18:01.136122 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:18:01.136070 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:18:02.936475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:18:02.936443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:18:08.321462 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:18:08.321430 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:18:30.319747 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:18:30.319705 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:18:42.012762 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:18:42.012689 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:18:48.497918 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:18:48.497889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:19:06.131049 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:19:06.131016 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:19:18.226042 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:19:18.226010 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:19:32.724141 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:19:32.724092 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:19:42.229300 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:19:42.229264 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:19:47.680960 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:19:47.680925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:19:51.822704 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:19:51.822676 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:20:01.120648 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:20:01.120618 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:20:10.311932 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:20:10.311848 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:21:33.257634 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:21:33.257599 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:22:01.229370 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:22:01.229300 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:22:10.234300 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:22:10.234264 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:22:23.515377 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:22:23.515340 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:22:37.018796 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:22:37.018758 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:22:46.515401 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:22:46.515363 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:22:56.019609 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:22:56.019567 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:23:06.420858 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:06.420782 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:23:15.847207 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:15.847170 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:23:27.416334 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:27.416300 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:23:35.019947 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:35.019915 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:23:43.817052 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:43.817016 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:23:53.134650 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:53.134619 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:23:55.616574 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:55.616531 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:23:56.851786 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:23:56.851756 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:24:10.726889 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:10.726856 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:24:14.256019 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:14.255986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:24:20.914910 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:20.914876 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:24:24.674285 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:24.674254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:24:33.815032 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:33.814954 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:24:41.716589 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.716554 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj"] Jul 21 17:24:41.719830 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.719813 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.722984 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.722966 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-003af18a-kserve-self-signed-certs\"" Jul 21 17:24:41.732947 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.732925 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj"] Jul 21 17:24:41.762255 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.762225 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:24:41.833837 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.833807 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-dshm\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.834007 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.833854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tls-certs\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.834007 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.833884 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-home\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.834007 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.833911 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9g7\" (UniqueName: \"kubernetes.io/projected/24a7bb50-6048-4cde-bd1a-02b7a58cd081-kube-api-access-dh9g7\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.834007 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.833932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tmp-dir\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.834007 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.833957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-model-cache\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935360 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935325 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-model-cache\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935390 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-dshm\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tls-certs\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-home\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935460 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9g7\" (UniqueName: \"kubernetes.io/projected/24a7bb50-6048-4cde-bd1a-02b7a58cd081-kube-api-access-dh9g7\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935482 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tmp-dir\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935785 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935718 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-model-cache\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935868 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tmp-dir\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.935934 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.935894 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-home\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.937679 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.937653 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-dshm\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.937837 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.937819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tls-certs\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:41.946069 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:41.946047 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9g7\" (UniqueName: \"kubernetes.io/projected/24a7bb50-6048-4cde-bd1a-02b7a58cd081-kube-api-access-dh9g7\") pod \"e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:42.028641 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:42.028610 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:42.162242 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:42.162217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj"] Jul 21 17:24:42.164243 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:24:42.164212 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24a7bb50_6048_4cde_bd1a_02b7a58cd081.slice/crio-da448b06e0d66e0359656504e5a4558eebfe6162950a5ebf0492acf0256c1fd3 WatchSource:0}: Error finding container da448b06e0d66e0359656504e5a4558eebfe6162950a5ebf0492acf0256c1fd3: Status 404 returned error can't find the container with id da448b06e0d66e0359656504e5a4558eebfe6162950a5ebf0492acf0256c1fd3 Jul 21 17:24:42.348321 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:42.348237 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" event={"ID":"24a7bb50-6048-4cde-bd1a-02b7a58cd081","Type":"ContainerStarted","Data":"40ab1603fa6af36e0d7f644adf65b353509827287b055cdb35a66ab13f787ae7"} Jul 21 17:24:42.348321 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:42.348275 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" event={"ID":"24a7bb50-6048-4cde-bd1a-02b7a58cd081","Type":"ContainerStarted","Data":"da448b06e0d66e0359656504e5a4558eebfe6162950a5ebf0492acf0256c1fd3"} Jul 21 17:24:42.367966 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:42.367912 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" podStartSLOduration=1.3678953090000001 podStartE2EDuration="1.367895309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:24:42.366971624 +0000 UTC m=+1350.481118621" watchObservedRunningTime="2026-07-21 17:24:42.367895309 +0000 UTC m=+1350.482042294" Jul 21 17:24:46.651645 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:46.651611 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:24:52.029551 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:52.029514 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:52.030000 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:52.029568 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:52.041741 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:52.041717 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:24:52.405342 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:24:52.405264 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:25:06.002061 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:06.002027 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:25:07.267567 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:07.267536 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj"] Jul 21 17:25:07.290513 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:07.290486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:25:10.128746 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:10.128712 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:25:22.287709 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.287631 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" podUID="24a7bb50-6048-4cde-bd1a-02b7a58cd081" containerName="main" containerID="cri-o://40ab1603fa6af36e0d7f644adf65b353509827287b055cdb35a66ab13f787ae7" gracePeriod=45 Jul 21 17:25:22.395065 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.395027 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" podUID="24a7bb50-6048-4cde-bd1a-02b7a58cd081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.29:8000/ready\": dial tcp 10.134.0.29:8000: connect: connection refused" Jul 21 17:25:22.531619 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.531594 2562 generic.go:356] "Generic (PLEG): container finished" podID="24a7bb50-6048-4cde-bd1a-02b7a58cd081" containerID="40ab1603fa6af36e0d7f644adf65b353509827287b055cdb35a66ab13f787ae7" exitCode=0 Jul 21 17:25:22.531732 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.531627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" event={"ID":"24a7bb50-6048-4cde-bd1a-02b7a58cd081","Type":"ContainerDied","Data":"40ab1603fa6af36e0d7f644adf65b353509827287b055cdb35a66ab13f787ae7"} Jul 21 17:25:22.531732 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.531653 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" event={"ID":"24a7bb50-6048-4cde-bd1a-02b7a58cd081","Type":"ContainerDied","Data":"da448b06e0d66e0359656504e5a4558eebfe6162950a5ebf0492acf0256c1fd3"} Jul 21 17:25:22.531732 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.531663 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da448b06e0d66e0359656504e5a4558eebfe6162950a5ebf0492acf0256c1fd3" Jul 21 17:25:22.533384 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.533368 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:25:22.588139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588055 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-dshm\") pod \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " Jul 21 17:25:22.588267 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588186 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tmp-dir\") pod \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " Jul 21 17:25:22.588267 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588224 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tls-certs\") pod \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " Jul 21 17:25:22.588379 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588268 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh9g7\" (UniqueName: \"kubernetes.io/projected/24a7bb50-6048-4cde-bd1a-02b7a58cd081-kube-api-access-dh9g7\") pod \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " Jul 21 17:25:22.588379 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588318 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-home\") pod \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " Jul 21 17:25:22.588379 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588342 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-model-cache\") pod \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\" (UID: \"24a7bb50-6048-4cde-bd1a-02b7a58cd081\") " Jul 21 17:25:22.588521 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588431 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24a7bb50-6048-4cde-bd1a-02b7a58cd081" (UID: "24a7bb50-6048-4cde-bd1a-02b7a58cd081"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:25:22.588573 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588523 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-home" (OuterVolumeSpecName: "home") pod "24a7bb50-6048-4cde-bd1a-02b7a58cd081" (UID: "24a7bb50-6048-4cde-bd1a-02b7a58cd081"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:25:22.588674 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588641 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-model-cache" (OuterVolumeSpecName: "model-cache") pod "24a7bb50-6048-4cde-bd1a-02b7a58cd081" (UID: "24a7bb50-6048-4cde-bd1a-02b7a58cd081"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:25:22.588743 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588726 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tmp-dir\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:25:22.588801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.588744 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-home\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:25:22.590424 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.590398 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24a7bb50-6048-4cde-bd1a-02b7a58cd081" (UID: "24a7bb50-6048-4cde-bd1a-02b7a58cd081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 17:25:22.590518 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.590429 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-dshm" (OuterVolumeSpecName: "dshm") pod "24a7bb50-6048-4cde-bd1a-02b7a58cd081" (UID: "24a7bb50-6048-4cde-bd1a-02b7a58cd081"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:25:22.590518 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.590437 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a7bb50-6048-4cde-bd1a-02b7a58cd081-kube-api-access-dh9g7" (OuterVolumeSpecName: "kube-api-access-dh9g7") pod "24a7bb50-6048-4cde-bd1a-02b7a58cd081" (UID: "24a7bb50-6048-4cde-bd1a-02b7a58cd081"). InnerVolumeSpecName "kube-api-access-dh9g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:25:22.689994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.689965 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24a7bb50-6048-4cde-bd1a-02b7a58cd081-tls-certs\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:25:22.689994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.689988 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh9g7\" (UniqueName: \"kubernetes.io/projected/24a7bb50-6048-4cde-bd1a-02b7a58cd081-kube-api-access-dh9g7\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:25:22.689994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.689998 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-model-cache\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:25:22.690214 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:22.690006 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24a7bb50-6048-4cde-bd1a-02b7a58cd081-dshm\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:25:23.536597 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:23.536566 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj" Jul 21 17:25:23.560994 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:23.560969 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj"] Jul 21 17:25:23.565261 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:23.565236 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-003af18a-kserve-76cfdc5479-hsfqj"] Jul 21 17:25:23.623547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:23.623520 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:25:24.377625 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:24.377586 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a7bb50-6048-4cde-bd1a-02b7a58cd081" path="/var/lib/kubelet/pods/24a7bb50-6048-4cde-bd1a-02b7a58cd081/volumes" Jul 21 17:25:30.323672 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:30.323633 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:25:44.327708 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:44.327677 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:25:55.824317 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:25:55.824283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:26:04.726202 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:26:04.726131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:26:15.423198 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:26:15.423165 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:26:24.223124 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:26:24.223070 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:26:34.126333 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:26:34.126300 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:26:47.926605 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:26:47.926566 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:26:56.027795 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:26:56.027761 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:27:05.635555 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:27:05.635526 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:27:17.147185 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:27:17.147154 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:27:22.894088 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:27:22.894051 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:27:27.521750 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:27:27.521716 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:27:34.526845 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:27:34.526769 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:27:50.131801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:27:50.131770 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:28:01.921302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:01.921262 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:28:11.421630 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:11.421589 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:28:13.476161 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:13.476091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:25.220320 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:25.220283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:28:31.688285 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:31.688253 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:31.760237 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:31.760201 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:34.815392 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:34.815362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:34.917426 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:34.917392 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:28:37.755071 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:37.755040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:37.857121 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:37.857071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:38.192750 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:38.192663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:44.724396 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:44.724361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:44.827668 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:44.827641 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:44.834773 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:44.834746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:28:45.064272 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:45.064190 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:45.392436 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:45.392360 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:45.546438 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:45.546404 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:47.462161 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:47.462126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:47.613721 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:47.613689 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:47.814906 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:47.814868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:48.058613 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:48.058568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:48.259835 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:48.259805 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:48.365664 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:48.365633 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:28:53.033263 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:28:53.033229 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:29:23.679163 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:23.679071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:26.930925 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:26.930883 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:29:28.094526 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:28.094490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:28.296390 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:28.296354 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:28.545232 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:28.545202 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:28.753117 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:28.753071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:28.855895 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:28.855816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:28.991078 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:28.991048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:29.319038 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:29.319009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:29.566852 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:29.566815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:31.794341 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:31.794301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:32.081995 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:32.081911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:32.308935 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:32.308902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:32.769517 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:32.769483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:32.865157 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:32.865127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:29:37.820760 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:29:37.820727 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:30:07.290937 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:07.290899 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:10.330639 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:10.330605 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:30:11.835856 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:11.835823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:12.010505 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:12.010469 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:12.242279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:12.242246 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:12.554984 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:12.554901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:12.818791 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:12.818710 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:12.937408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:12.937376 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:14.993961 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:14.993933 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:15.249862 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:15.249784 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:15.395407 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:15.395375 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:15.923488 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:15.923455 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:16.087922 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:16.087891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:16.187999 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:16.187913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:21.226274 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:21.226240 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:30:51.373376 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:51.373293 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:54.417116 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:54.417080 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:30:55.839793 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:55.839751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:57.087352 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:57.087319 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:57.305437 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:57.305403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:57.520585 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:57.520550 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:57.913304 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:57.913212 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:30:58.083318 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:30:58.083289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:00.173155 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:00.173092 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:00.267931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:00.267902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:00.353084 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:00.353052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:00.452082 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:00.452006 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:01.204895 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:01.204858 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:01.300864 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:01.300829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:05.532788 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:05.532756 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:31:12.395482 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:12.395454 2562 scope.go:117] "RemoveContainer" containerID="40ab1603fa6af36e0d7f644adf65b353509827287b055cdb35a66ab13f787ae7" Jul 21 17:31:17.907301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:17.907269 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:18.007808 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:18.007779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:23.325039 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:23.325001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:23.425997 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:23.425950 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:35.591639 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:35.591603 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:38.514855 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:38.514820 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:31:40.148998 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:40.148954 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:40.314540 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:40.314504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:40.545459 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:40.545429 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:41.026962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:41.026929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:41.111989 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:41.111953 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:41.213774 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:41.213741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:42.708880 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:42.708842 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:50.841149 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:50.841098 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:50.948026 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:50.947993 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:51.049278 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:51.049233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:51.151223 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:51.151141 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:31:51.575730 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:31:51.575693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:17.546037 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:17.545995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:25.326640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:25.326604 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:25.491025 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:25.490991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:25.641853 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:25.641756 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:41.536509 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:41.536472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:48.233298 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:48.233264 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:48.343462 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:48.343426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:48.443122 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:48.443064 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:58.978839 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:58.978804 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:32:59.771590 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:32:59.771552 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:07.729166 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:07.729131 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:07.831949 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:07.831911 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:07.943182 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:07.943150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:08.053079 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:08.052996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:08.840437 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:08.840403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:08.936867 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:08.936838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:09.045515 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:09.045478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:10.318466 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:10.318433 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:33:10.674857 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:10.674764 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:17.706845 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:17.706811 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:17.807883 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:17.807849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:18.431894 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:18.431860 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:20.421239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:20.421205 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:33:25.556912 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:25.556867 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:25.677925 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:25.677880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:25.821038 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:25.820946 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:26.202178 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:26.202074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:26.370838 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:26.370803 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:29.498639 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:29.498600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:29.595822 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:29.595790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:29.747661 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:29.747615 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:29.864344 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:29.864255 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:30.040999 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:30.040967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:30.488997 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:30.488964 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:30.591494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:30.591460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:32.644993 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:32.644964 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:33:33.991505 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:33.991420 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:34.605492 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:34.605457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:34.704089 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:34.704052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:35.120947 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:35.120901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:41.687279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:41.687250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:41.787181 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:41.787146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:52.494944 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:52.494910 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:55.423323 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:55.423291 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:33:57.095590 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:57.095549 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:33:57.500615 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:33:57.500575 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:04.233723 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:04.233679 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:04.332952 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:04.332909 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:04.542719 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:04.542634 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:04.802481 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:04.802401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:04.975238 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:04.975205 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:05.111008 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:05.110918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:05.240132 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:05.240068 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:11.869654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:11.869619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:11.966506 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:11.966466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:12.074725 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:12.074691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:12.186159 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:12.186063 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:19.184870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:19.184834 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:19.279445 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:19.279415 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:26.002129 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:26.002071 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:28.001946 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:28.001913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:29.777362 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:29.777325 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:29.878494 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:29.878459 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:30.095038 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:30.094944 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:30.470229 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:30.470186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:30.793282 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:30.793180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:30.899290 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:30.899258 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:34.279566 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:34.279538 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:36.531189 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:36.531149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:34:52.221554 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.221521 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t"] Jul 21 17:34:52.221929 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.221887 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a7bb50-6048-4cde-bd1a-02b7a58cd081" containerName="main" Jul 21 17:34:52.221929 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.221897 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a7bb50-6048-4cde-bd1a-02b7a58cd081" containerName="main" Jul 21 17:34:52.222004 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.221965 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a7bb50-6048-4cde-bd1a-02b7a58cd081" containerName="main" Jul 21 17:34:52.224982 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.224966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.228287 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.228262 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-28b3afa2\"/\"kube-root-ca.crt\"" Jul 21 17:34:52.229253 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.229232 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-28b3afa2\"/\"openshift-service-ca.crt\"" Jul 21 17:34:52.229253 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.229249 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-28b3afa2\"/\"default-dockercfg-wr62g\"" Jul 21 17:34:52.229426 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.229288 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-28b3afa2\"/\"e2e-lifecycle-model-28b3afa2-kserve-self-signed-certs\"" Jul 21 17:34:52.242466 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.242442 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t"] Jul 21 17:34:52.268587 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.268559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:34:52.371971 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.371940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-model-cache\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.371971 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.371976 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tmp-dir\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.372198 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.371998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tls-certs\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.372198 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.372157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-home\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.372307 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.372199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvzv\" (UniqueName: \"kubernetes.io/projected/a8632d52-c6fa-42ee-9b1f-f02594f091f9-kube-api-access-jdvzv\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.372361 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.372319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-dshm\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473071 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.472997 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-home\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473071 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvzv\" (UniqueName: \"kubernetes.io/projected/a8632d52-c6fa-42ee-9b1f-f02594f091f9-kube-api-access-jdvzv\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473316 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473087 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-dshm\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473316 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-model-cache\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473316 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tmp-dir\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473316 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tls-certs\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473401 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-home\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-model-cache\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.473614 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.473555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tmp-dir\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.475291 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.475264 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-dshm\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.475661 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.475641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tls-certs\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.482504 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.482483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvzv\" (UniqueName: \"kubernetes.io/projected/a8632d52-c6fa-42ee-9b1f-f02594f091f9-kube-api-access-jdvzv\") pod \"e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.534525 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.534498 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:34:52.666287 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:52.666257 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t"] Jul 21 17:34:52.668353 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:34:52.668323 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8632d52_c6fa_42ee_9b1f_f02594f091f9.slice/crio-0209cabaf5874ccea5212db40794d7cbdc51e156dd5112145360c68bb1095962 WatchSource:0}: Error finding container 0209cabaf5874ccea5212db40794d7cbdc51e156dd5112145360c68bb1095962: Status 404 returned error can't find the container with id 0209cabaf5874ccea5212db40794d7cbdc51e156dd5112145360c68bb1095962 Jul 21 17:34:53.124262 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:53.124170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" event={"ID":"a8632d52-c6fa-42ee-9b1f-f02594f091f9","Type":"ContainerStarted","Data":"34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62"} Jul 21 17:34:53.124262 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:53.124210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" event={"ID":"a8632d52-c6fa-42ee-9b1f-f02594f091f9","Type":"ContainerStarted","Data":"0209cabaf5874ccea5212db40794d7cbdc51e156dd5112145360c68bb1095962"} Jul 21 17:34:55.121436 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:55.121377 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" podStartSLOduration=3.121359743 podStartE2EDuration="3.121359743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:34:53.153305026 +0000 UTC m=+1961.267452011" watchObservedRunningTime="2026-07-21 17:34:55.121359743 +0000 UTC m=+1963.235506776" Jul 21 17:34:55.122032 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:34:55.122015 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:35:02.535621 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:02.535576 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:35:02.536194 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:02.535635 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:35:02.548268 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:02.548243 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:35:03.180497 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:03.180468 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:35:04.189567 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:04.189527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:35:09.430054 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:09.430018 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:35:27.121212 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:27.121176 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:35:28.431647 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:28.431616 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:35:32.908586 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:32.908558 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:35:35.144075 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:35.144039 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:35:35.443688 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:35.443609 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:35:40.099933 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:40.099897 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t"] Jul 21 17:35:40.180776 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:40.180731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:35:44.947709 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:44.947663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:35:45.065704 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:45.065669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:35:55.119814 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.119749 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" podUID="a8632d52-c6fa-42ee-9b1f-f02594f091f9" containerName="main" containerID="cri-o://34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62" gracePeriod=45 Jul 21 17:35:55.365273 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.365249 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:35:55.411445 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.411367 2562 generic.go:356] "Generic (PLEG): container finished" podID="a8632d52-c6fa-42ee-9b1f-f02594f091f9" containerID="34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62" exitCode=0 Jul 21 17:35:55.411445 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.411402 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" Jul 21 17:35:55.411636 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.411443 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" event={"ID":"a8632d52-c6fa-42ee-9b1f-f02594f091f9","Type":"ContainerDied","Data":"34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62"} Jul 21 17:35:55.411636 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.411483 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t" event={"ID":"a8632d52-c6fa-42ee-9b1f-f02594f091f9","Type":"ContainerDied","Data":"0209cabaf5874ccea5212db40794d7cbdc51e156dd5112145360c68bb1095962"} Jul 21 17:35:55.411636 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.411501 2562 scope.go:117] "RemoveContainer" containerID="34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62" Jul 21 17:35:55.420051 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.420034 2562 scope.go:117] "RemoveContainer" containerID="34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62" Jul 21 17:35:55.420331 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:35:55.420314 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62\": container with ID starting with 34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62 not found: ID does not exist" containerID="34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62" Jul 21 17:35:55.420411 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.420344 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62"} err="failed to get container status \"34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62\": rpc error: code = NotFound desc = could not find container \"34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62\": container with ID starting with 34367640e49278180a4dbb2dc16e96ee5b5e93551288954050547120ea161a62 not found: ID does not exist" Jul 21 17:35:55.431685 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.431666 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdvzv\" (UniqueName: \"kubernetes.io/projected/a8632d52-c6fa-42ee-9b1f-f02594f091f9-kube-api-access-jdvzv\") pod \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " Jul 21 17:35:55.431777 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.431712 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-home\") pod \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " Jul 21 17:35:55.431777 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.431744 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-dshm\") pod \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " Jul 21 17:35:55.431880 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.431802 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-model-cache\") pod \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " Jul 21 17:35:55.431880 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.431836 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tmp-dir\") pod \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " Jul 21 17:35:55.431983 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.431885 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tls-certs\") pod \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\" (UID: \"a8632d52-c6fa-42ee-9b1f-f02594f091f9\") " Jul 21 17:35:55.432095 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.432072 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-home" (OuterVolumeSpecName: "home") pod "a8632d52-c6fa-42ee-9b1f-f02594f091f9" (UID: "a8632d52-c6fa-42ee-9b1f-f02594f091f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:35:55.432187 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.432089 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a8632d52-c6fa-42ee-9b1f-f02594f091f9" (UID: "a8632d52-c6fa-42ee-9b1f-f02594f091f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:35:55.432246 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.432208 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a8632d52-c6fa-42ee-9b1f-f02594f091f9" (UID: "a8632d52-c6fa-42ee-9b1f-f02594f091f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:35:55.432308 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.432262 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-model-cache\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:35:55.432308 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.432277 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-home\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:35:55.433952 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.433928 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-dshm" (OuterVolumeSpecName: "dshm") pod "a8632d52-c6fa-42ee-9b1f-f02594f091f9" (UID: "a8632d52-c6fa-42ee-9b1f-f02594f091f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:35:55.434042 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.433983 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a8632d52-c6fa-42ee-9b1f-f02594f091f9" (UID: "a8632d52-c6fa-42ee-9b1f-f02594f091f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 17:35:55.434124 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.434087 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8632d52-c6fa-42ee-9b1f-f02594f091f9-kube-api-access-jdvzv" (OuterVolumeSpecName: "kube-api-access-jdvzv") pod "a8632d52-c6fa-42ee-9b1f-f02594f091f9" (UID: "a8632d52-c6fa-42ee-9b1f-f02594f091f9"). InnerVolumeSpecName "kube-api-access-jdvzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:35:55.532957 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.532931 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-dshm\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:35:55.533076 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.532970 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tmp-dir\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:35:55.533076 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.532980 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8632d52-c6fa-42ee-9b1f-f02594f091f9-tls-certs\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:35:55.533076 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.532989 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvzv\" (UniqueName: \"kubernetes.io/projected/a8632d52-c6fa-42ee-9b1f-f02594f091f9-kube-api-access-jdvzv\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:35:55.739661 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.739630 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t"] Jul 21 17:35:55.745793 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:55.745759 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-28b3afa2/e2e-lifecycle-model-28b3afa2-kserve-599c5f5f99-m6d8t"] Jul 21 17:35:56.376928 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:35:56.376891 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8632d52-c6fa-42ee-9b1f-f02594f091f9" path="/var/lib/kubelet/pods/a8632d52-c6fa-42ee-9b1f-f02594f091f9/volumes" Jul 21 17:36:16.936221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:16.936183 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:17.161514 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:17.161484 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:18.909628 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:18.909594 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:19.302850 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:19.302817 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:19.402367 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:19.402336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:24.788242 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:24.788206 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:25.503349 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:25.503311 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:41.633301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:41.633267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:41.740056 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:41.740015 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:41.966064 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:41.965980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:42.233509 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:42.233472 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:43.310295 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:43.310260 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:43.403905 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:43.403871 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:43.503968 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:43.503941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:44.231810 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:44.231772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:44.329592 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:44.329555 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:44.425678 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:44.425644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:45.840808 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:45.840774 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:36:45.967542 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:45.967478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:52.350408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:52.350361 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:52.452288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:52.452259 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:53.051961 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:53.051927 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:36:55.976773 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:36:55.976741 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:37:01.317357 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:01.317317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:01.481223 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:01.481182 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:01.620949 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:01.620874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:12.112924 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:12.112887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:12.521511 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:12.521478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:33.448480 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:33.448446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:42.332165 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:42.332120 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:42.449837 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:42.449807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:42.673906 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:42.673836 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:43.036805 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:43.036762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:43.260415 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:43.260379 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:45.844541 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:45.844508 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:45.946438 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:45.946411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:46.104826 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:46.104743 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:46.259448 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:46.259413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:46.828343 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:46.828308 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:46.936684 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:46.936652 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:50.327092 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:50.327060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:50.655069 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:50.654993 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:51.012999 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:51.012958 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:51.545505 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:51.545474 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:51.552478 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:51.552456 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:37:54.020332 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:54.020293 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:54.140065 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:54.140017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:54.251552 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:54.251522 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:54.529857 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:54.529826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:55.055406 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:55.055362 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:55.155602 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:55.155566 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:37:58.470926 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:37:58.470888 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:38:03.730579 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:03.730500 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:38:04.137884 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.137810 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw"] Jul 21 17:38:04.138183 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.138170 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8632d52-c6fa-42ee-9b1f-f02594f091f9" containerName="main" Jul 21 17:38:04.138237 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.138184 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8632d52-c6fa-42ee-9b1f-f02594f091f9" containerName="main" Jul 21 17:38:04.138283 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.138246 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8632d52-c6fa-42ee-9b1f-f02594f091f9" containerName="main" Jul 21 17:38:04.141059 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.141032 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.144297 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.144277 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d0798\"/\"default-dockercfg-bldls\"" Jul 21 17:38:04.145009 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.144994 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d0798\"/\"openshift-service-ca.crt\"" Jul 21 17:38:04.145160 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.145139 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d0798\"/\"test-model-2d0798-kserve-self-signed-certs\"" Jul 21 17:38:04.145227 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.145151 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d0798\"/\"kube-root-ca.crt\"" Jul 21 17:38:04.153562 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.153541 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw"] Jul 21 17:38:04.175380 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.175356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:38:04.245449 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.245423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55755c4-a925-4e04-9722-68f289ceebdc-tls-certs\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.245559 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.245454 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-home\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.245559 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.245489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-model-cache\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.245559 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.245552 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdtbv\" (UniqueName: \"kubernetes.io/projected/f55755c4-a925-4e04-9722-68f289ceebdc-kube-api-access-pdtbv\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.245679 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.245585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-dshm\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.245679 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.245664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-tmp-dir\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.346538 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55755c4-a925-4e04-9722-68f289ceebdc-tls-certs\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.346736 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-home\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.346736 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-model-cache\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.346736 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdtbv\" (UniqueName: \"kubernetes.io/projected/f55755c4-a925-4e04-9722-68f289ceebdc-kube-api-access-pdtbv\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.346736 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-dshm\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.346736 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-tmp-dir\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.346986 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346968 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-home\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.347048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.346960 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-model-cache\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.347048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.347012 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-tmp-dir\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.348784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.348756 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-dshm\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.349027 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.349010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55755c4-a925-4e04-9722-68f289ceebdc-tls-certs\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.360825 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.360804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdtbv\" (UniqueName: \"kubernetes.io/projected/f55755c4-a925-4e04-9722-68f289ceebdc-kube-api-access-pdtbv\") pod \"test-model-2d0798-kserve-d65c4dcc7-tp2nw\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.450381 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.450303 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:04.585516 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.585494 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw"] Jul 21 17:38:04.587288 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:38:04.587260 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf55755c4_a925_4e04_9722_68f289ceebdc.slice/crio-2be89b8aea319b17e6885e509d96f62062fdfc2941524140c4abe434558917da WatchSource:0}: Error finding container 2be89b8aea319b17e6885e509d96f62062fdfc2941524140c4abe434558917da: Status 404 returned error can't find the container with id 2be89b8aea319b17e6885e509d96f62062fdfc2941524140c4abe434558917da Jul 21 17:38:04.997459 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.997418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" event={"ID":"f55755c4-a925-4e04-9722-68f289ceebdc","Type":"ContainerStarted","Data":"9885ca8129f4486b33be18061098ea2f2b25e8070489a1b1bf97debf4564da65"} Jul 21 17:38:04.997459 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:04.997461 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" event={"ID":"f55755c4-a925-4e04-9722-68f289ceebdc","Type":"ContainerStarted","Data":"2be89b8aea319b17e6885e509d96f62062fdfc2941524140c4abe434558917da"} Jul 21 17:38:05.022618 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:05.022571 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" podStartSLOduration=1.022557312 podStartE2EDuration="1.022557312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:38:05.019719632 +0000 UTC m=+2153.133866617" watchObservedRunningTime="2026-07-21 17:38:05.022557312 +0000 UTC m=+2153.136704296" Jul 21 17:38:07.933791 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:07.933761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:38:08.107891 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:08.107857 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:38:13.867772 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:13.867714 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:38:14.451124 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:14.451076 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:14.451337 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:14.451141 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:14.463606 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:14.463577 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:15.053382 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:15.053347 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:38:17.979856 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:17.979816 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:38:18.082171 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:18.082132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:38:40.270607 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:40.270568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:38:45.638438 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:38:45.638399 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:39:18.190990 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:18.190957 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw"] Jul 21 17:39:18.273556 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:18.273529 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:19.101328 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:19.101294 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:19.973422 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:19.973387 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:20.073463 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:20.073429 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:23.424341 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:23.424306 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:39:29.800048 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:29.800006 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:33.210528 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.210488 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" podUID="f55755c4-a925-4e04-9722-68f289ceebdc" containerName="main" containerID="cri-o://9885ca8129f4486b33be18061098ea2f2b25e8070489a1b1bf97debf4564da65" gracePeriod=45 Jul 21 17:39:33.415236 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.415206 2562 generic.go:356] "Generic (PLEG): container finished" podID="f55755c4-a925-4e04-9722-68f289ceebdc" containerID="9885ca8129f4486b33be18061098ea2f2b25e8070489a1b1bf97debf4564da65" exitCode=0 Jul 21 17:39:33.415381 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.415262 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" event={"ID":"f55755c4-a925-4e04-9722-68f289ceebdc","Type":"ContainerDied","Data":"9885ca8129f4486b33be18061098ea2f2b25e8070489a1b1bf97debf4564da65"} Jul 21 17:39:33.449788 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.449764 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:39:33.549427 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549393 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-home\") pod \"f55755c4-a925-4e04-9722-68f289ceebdc\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " Jul 21 17:39:33.549604 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549455 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-model-cache\") pod \"f55755c4-a925-4e04-9722-68f289ceebdc\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " Jul 21 17:39:33.549604 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549486 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-dshm\") pod \"f55755c4-a925-4e04-9722-68f289ceebdc\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " Jul 21 17:39:33.549604 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549508 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-tmp-dir\") pod \"f55755c4-a925-4e04-9722-68f289ceebdc\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " Jul 21 17:39:33.549604 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549560 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55755c4-a925-4e04-9722-68f289ceebdc-tls-certs\") pod \"f55755c4-a925-4e04-9722-68f289ceebdc\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " Jul 21 17:39:33.549604 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549597 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdtbv\" (UniqueName: \"kubernetes.io/projected/f55755c4-a925-4e04-9722-68f289ceebdc-kube-api-access-pdtbv\") pod \"f55755c4-a925-4e04-9722-68f289ceebdc\" (UID: \"f55755c4-a925-4e04-9722-68f289ceebdc\") " Jul 21 17:39:33.549915 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549697 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-home" (OuterVolumeSpecName: "home") pod "f55755c4-a925-4e04-9722-68f289ceebdc" (UID: "f55755c4-a925-4e04-9722-68f289ceebdc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:39:33.549915 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549738 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-model-cache" (OuterVolumeSpecName: "model-cache") pod "f55755c4-a925-4e04-9722-68f289ceebdc" (UID: "f55755c4-a925-4e04-9722-68f289ceebdc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:39:33.549915 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549786 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f55755c4-a925-4e04-9722-68f289ceebdc" (UID: "f55755c4-a925-4e04-9722-68f289ceebdc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:39:33.550063 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549925 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-model-cache\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:39:33.550063 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549940 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-tmp-dir\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:39:33.550063 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.549948 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-home\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:39:33.551696 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.551674 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55755c4-a925-4e04-9722-68f289ceebdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f55755c4-a925-4e04-9722-68f289ceebdc" (UID: "f55755c4-a925-4e04-9722-68f289ceebdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 17:39:33.551785 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.551677 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55755c4-a925-4e04-9722-68f289ceebdc-kube-api-access-pdtbv" (OuterVolumeSpecName: "kube-api-access-pdtbv") pod "f55755c4-a925-4e04-9722-68f289ceebdc" (UID: "f55755c4-a925-4e04-9722-68f289ceebdc"). InnerVolumeSpecName "kube-api-access-pdtbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:39:33.551785 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.551731 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-dshm" (OuterVolumeSpecName: "dshm") pod "f55755c4-a925-4e04-9722-68f289ceebdc" (UID: "f55755c4-a925-4e04-9722-68f289ceebdc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:39:33.650962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.650937 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55755c4-a925-4e04-9722-68f289ceebdc-tls-certs\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:39:33.650962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.650959 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdtbv\" (UniqueName: \"kubernetes.io/projected/f55755c4-a925-4e04-9722-68f289ceebdc-kube-api-access-pdtbv\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:39:33.651140 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:33.650968 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f55755c4-a925-4e04-9722-68f289ceebdc-dshm\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:39:34.345843 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:34.345800 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:34.421301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:34.421271 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" event={"ID":"f55755c4-a925-4e04-9722-68f289ceebdc","Type":"ContainerDied","Data":"2be89b8aea319b17e6885e509d96f62062fdfc2941524140c4abe434558917da"} Jul 21 17:39:34.421301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:34.421297 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw" Jul 21 17:39:34.421499 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:34.421319 2562 scope.go:117] "RemoveContainer" containerID="9885ca8129f4486b33be18061098ea2f2b25e8070489a1b1bf97debf4564da65" Jul 21 17:39:34.443368 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:34.443344 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw"] Jul 21 17:39:34.450513 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:34.450490 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2d0798/test-model-2d0798-kserve-d65c4dcc7-tp2nw"] Jul 21 17:39:34.746788 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:34.746752 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:36.377575 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:36.377497 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55755c4-a925-4e04-9722-68f289ceebdc" path="/var/lib/kubelet/pods/f55755c4-a925-4e04-9722-68f289ceebdc/volumes" Jul 21 17:39:38.222603 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:38.222571 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:39:41.316620 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:41.316582 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:41.398783 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:41.398731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:41.682598 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:41.682518 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:41.879585 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:41.879548 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:42.575267 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:42.575222 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:42.674953 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:42.674925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:52.648995 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:52.648960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:55.551082 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:55.551050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:39:57.063609 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:57.063570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:39:57.443138 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:39:57.443035 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:04.221945 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:04.221903 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:04.334524 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:04.334488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:04.593580 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:04.593498 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:04.835258 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:04.835221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:05.196207 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:05.196171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:05.574784 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:05.574741 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:06.002466 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:06.002427 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:08.564768 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:08.564713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:08.685801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:08.685768 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:08.894965 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:08.894866 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:09.185652 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:09.185573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:09.575861 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:09.575829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:09.673455 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:09.673422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:13.037946 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:13.037907 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:13.429974 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:13.429894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:40:13.528073 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:13.528037 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:13.910408 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:13.910371 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:14.197014 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:14.196932 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:16.605328 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:16.605294 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:16.705125 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:16.705074 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:16.826655 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:16.826620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:16.976562 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:16.976528 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:17.345557 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:17.345453 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:17.441803 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:17.441762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:21.081967 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:21.081939 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:23.927460 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:23.927424 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:40:29.754335 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:29.754300 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:29.863161 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:29.863078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:35.652844 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:35.652794 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:35.720954 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:35.720930 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:38.269768 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:38.269728 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:38.370641 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:38.370602 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:39.679171 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:39.679133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:39.780419 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:39.780384 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:44.971459 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:44.971419 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:49.149016 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:49.148976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:49.522849 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:49.522815 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:50.926066 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:50.926034 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:40:56.019979 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:56.019940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:56.094173 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:56.094139 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:56.303130 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:56.303028 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:56.604665 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:56.604576 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:57.239973 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:57.239938 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:57.338266 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:57.338234 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:40:58.460811 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:40:58.460773 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:12.184028 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:12.183958 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:16.738404 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:16.738348 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:17.133176 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:17.133072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:17.217847 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:17.217814 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:41:19.970425 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:19.970393 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:20.070880 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:20.070847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:23.336282 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:23.336247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:23.439726 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:23.439684 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:23.610446 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:23.610364 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:23.836865 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:23.836822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:24.174085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:24.174041 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:24.522878 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:24.522844 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:24.788060 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:24.787992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:27.981379 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:27.981345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:28.069002 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:28.068973 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:28.189710 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:28.189675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:28.458781 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:28.458695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:28.842651 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:28.842613 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:28.942927 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:28.942893 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:32.471742 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:32.471712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:32.805661 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:32.805578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:33.108954 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:33.108838 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:33.409507 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:33.409425 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:35.519476 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:35.519444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:41:36.041854 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:36.041820 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:36.123493 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:36.123460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:36.206972 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:36.206937 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:36.533280 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:36.533248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:36.753828 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:36.753791 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:36.855294 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:36.855213 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:40.514749 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:40.514716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:41.264521 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:41.264466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:46.414428 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:46.414398 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:41:49.703352 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:49.703310 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:49.803516 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:49.803483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:57.727002 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:57.726968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:57.828743 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:57.828709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:41:58.524178 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:41:58.524145 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:42:02.208209 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:02.208176 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:42:38.747213 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.747152 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4"] Jul 21 17:42:38.747625 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.747560 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55755c4-a925-4e04-9722-68f289ceebdc" containerName="main" Jul 21 17:42:38.747625 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.747573 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55755c4-a925-4e04-9722-68f289ceebdc" containerName="main" Jul 21 17:42:38.747698 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.747641 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55755c4-a925-4e04-9722-68f289ceebdc" containerName="main" Jul 21 17:42:38.750406 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.750390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.753973 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.753949 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a36a6\"/\"kube-root-ca.crt\"" Jul 21 17:42:38.754134 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.753993 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a36a6\"/\"auth-test-model-5a36a6-kserve-self-signed-certs\"" Jul 21 17:42:38.754134 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.754054 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5a36a6\"/\"default-dockercfg-w2h92\"" Jul 21 17:42:38.754134 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.754080 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5a36a6\"/\"openshift-service-ca.crt\"" Jul 21 17:42:38.762696 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.762677 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4"] Jul 21 17:42:38.846287 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.846254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:42:38.872075 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.872048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-model-cache\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.872226 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.872081 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljl6w\" (UniqueName: \"kubernetes.io/projected/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-kube-api-access-ljl6w\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.872226 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.872176 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tmp-dir\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.872226 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.872208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-dshm\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.872349 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.872230 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tls-certs\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.872349 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.872251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-home\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.972725 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.972690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-model-cache\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.972725 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.972733 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljl6w\" (UniqueName: \"kubernetes.io/projected/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-kube-api-access-ljl6w\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.972919 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.972802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tmp-dir\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.972919 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.972836 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-dshm\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.972999 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.972957 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tls-certs\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.973052 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.973013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-home\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.973189 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.973169 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-model-cache\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.973308 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.973284 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tmp-dir\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.973381 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.973366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-home\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.975136 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.975116 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-dshm\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.975359 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.975339 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tls-certs\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:38.983026 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:38.983005 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljl6w\" (UniqueName: \"kubernetes.io/projected/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-kube-api-access-ljl6w\") pod \"auth-test-model-5a36a6-kserve-76f9cc4879-jvll4\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:39.059844 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:39.059768 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:39.193475 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:39.193450 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4"] Jul 21 17:42:39.195879 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:42:39.195850 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f8d78e1_2603_47b7_b6d5_248b8a2627ed.slice/crio-f43fc1036365d592d395a284b10a8ba575bba3223fd69f7fb39b90d987285c39 WatchSource:0}: Error finding container f43fc1036365d592d395a284b10a8ba575bba3223fd69f7fb39b90d987285c39: Status 404 returned error can't find the container with id f43fc1036365d592d395a284b10a8ba575bba3223fd69f7fb39b90d987285c39 Jul 21 17:42:39.267399 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:39.267358 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" event={"ID":"2f8d78e1-2603-47b7-b6d5-248b8a2627ed","Type":"ContainerStarted","Data":"0c4d5a7d6e76bac53af5351044dfc49409b2c7fead017d8e261194341354e2a3"} Jul 21 17:42:39.267550 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:39.267414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" event={"ID":"2f8d78e1-2603-47b7-b6d5-248b8a2627ed","Type":"ContainerStarted","Data":"f43fc1036365d592d395a284b10a8ba575bba3223fd69f7fb39b90d987285c39"} Jul 21 17:42:39.291601 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:39.291555 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" podStartSLOduration=1.291538147 podStartE2EDuration="1.291538147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:42:39.29091495 +0000 UTC m=+2427.405061936" watchObservedRunningTime="2026-07-21 17:42:39.291538147 +0000 UTC m=+2427.405685132" Jul 21 17:42:42.343955 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:42.343919 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:42:49.060556 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:49.060504 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:49.060963 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:49.060573 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:49.072720 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:49.072696 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:49.323151 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:49.323071 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:42:57.922909 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:42:57.922861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:43:09.557582 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:09.557547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:43:17.324074 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:17.324035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:43:20.676625 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:20.676594 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:43:31.163089 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:31.163058 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:43:36.025539 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:36.025500 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:43:54.026379 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:54.026330 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:43:56.672198 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:56.672162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:43:56.772902 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:43:56.772868 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:44:08.724352 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:08.724256 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:44:20.019704 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:20.019667 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:44:34.322849 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:34.322810 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:44:38.996623 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:38.996590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:44:39.736095 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:39.736062 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4"] Jul 21 17:44:40.528279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:40.528247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:44:50.521377 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:50.521342 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:44:52.336854 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.336810 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b"] Jul 21 17:44:52.339898 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.339880 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.342843 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.342819 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6fedc9\"/\"sub-test-model-6fedc9-kserve-self-signed-certs\"" Jul 21 17:44:52.342984 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.342853 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6fedc9\"/\"openshift-service-ca.crt\"" Jul 21 17:44:52.342984 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.342861 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6fedc9\"/\"default-dockercfg-qsqgm\"" Jul 21 17:44:52.342984 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.342829 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6fedc9\"/\"kube-root-ca.crt\"" Jul 21 17:44:52.353462 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.353435 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b"] Jul 21 17:44:52.365249 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.365224 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-dshm\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.365411 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.365297 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tls-certs\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.365411 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.365322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m5rm\" (UniqueName: \"kubernetes.io/projected/74b40d76-2508-44c0-9f6a-1e5a19202bf4-kube-api-access-6m5rm\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.365411 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.365366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-model-cache\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.365588 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.365461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tmp-dir\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.365588 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.365499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-home\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.371389 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.371342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:44:52.465914 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.465879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-model-cache\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.465914 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.465924 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tmp-dir\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.466216 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.465942 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-home\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.466216 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.465969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-dshm\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.466216 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.466018 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tls-certs\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.466216 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.466040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m5rm\" (UniqueName: \"kubernetes.io/projected/74b40d76-2508-44c0-9f6a-1e5a19202bf4-kube-api-access-6m5rm\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.466468 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.466443 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-model-cache\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.466530 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.466483 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-home\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.466575 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.466546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tmp-dir\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.468422 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.468397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-dshm\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.468698 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.468680 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tls-certs\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.475239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.475216 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m5rm\" (UniqueName: \"kubernetes.io/projected/74b40d76-2508-44c0-9f6a-1e5a19202bf4-kube-api-access-6m5rm\") pod \"sub-test-model-6fedc9-kserve-6dfb958d85-2b94b\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.650678 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.650582 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:44:52.787518 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.787485 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b"] Jul 21 17:44:52.790668 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:44:52.790642 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74b40d76_2508_44c0_9f6a_1e5a19202bf4.slice/crio-fac2e5ad2496413fca81c77b13e9a76d93465bcf5032bed20196dcd468d3651a WatchSource:0}: Error finding container fac2e5ad2496413fca81c77b13e9a76d93465bcf5032bed20196dcd468d3651a: Status 404 returned error can't find the container with id fac2e5ad2496413fca81c77b13e9a76d93465bcf5032bed20196dcd468d3651a Jul 21 17:44:52.896708 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.896675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" event={"ID":"74b40d76-2508-44c0-9f6a-1e5a19202bf4","Type":"ContainerStarted","Data":"31b4a4c2bc98f314ddb8118a5b709ce44a84584d6b479bb6bbb71fc2f2f1013f"} Jul 21 17:44:52.896708 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.896721 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" event={"ID":"74b40d76-2508-44c0-9f6a-1e5a19202bf4","Type":"ContainerStarted","Data":"fac2e5ad2496413fca81c77b13e9a76d93465bcf5032bed20196dcd468d3651a"} Jul 21 17:44:52.917956 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:52.917826 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" podStartSLOduration=0.917806907 podStartE2EDuration="917.806907ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:44:52.917541467 +0000 UTC m=+2561.031688453" watchObservedRunningTime="2026-07-21 17:44:52.917806907 +0000 UTC m=+2561.031953896" Jul 21 17:44:54.755980 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:54.755906 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" podUID="2f8d78e1-2603-47b7-b6d5-248b8a2627ed" containerName="main" containerID="cri-o://0c4d5a7d6e76bac53af5351044dfc49409b2c7fead017d8e261194341354e2a3" gracePeriod=45 Jul 21 17:44:54.908998 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:54.908968 2562 generic.go:356] "Generic (PLEG): container finished" podID="2f8d78e1-2603-47b7-b6d5-248b8a2627ed" containerID="0c4d5a7d6e76bac53af5351044dfc49409b2c7fead017d8e261194341354e2a3" exitCode=0 Jul 21 17:44:54.909199 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:54.909043 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" event={"ID":"2f8d78e1-2603-47b7-b6d5-248b8a2627ed","Type":"ContainerDied","Data":"0c4d5a7d6e76bac53af5351044dfc49409b2c7fead017d8e261194341354e2a3"} Jul 21 17:44:55.009975 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.009917 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:44:55.088367 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088338 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-model-cache\") pod \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " Jul 21 17:44:55.088367 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088366 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-dshm\") pod \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " Jul 21 17:44:55.088586 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088419 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tmp-dir\") pod \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " Jul 21 17:44:55.088586 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088437 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tls-certs\") pod \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " Jul 21 17:44:55.088586 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088464 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-home\") pod \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " Jul 21 17:44:55.088586 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088512 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljl6w\" (UniqueName: \"kubernetes.io/projected/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-kube-api-access-ljl6w\") pod \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\" (UID: \"2f8d78e1-2603-47b7-b6d5-248b8a2627ed\") " Jul 21 17:44:55.088795 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088604 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f8d78e1-2603-47b7-b6d5-248b8a2627ed" (UID: "2f8d78e1-2603-47b7-b6d5-248b8a2627ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:44:55.088795 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088713 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-home" (OuterVolumeSpecName: "home") pod "2f8d78e1-2603-47b7-b6d5-248b8a2627ed" (UID: "2f8d78e1-2603-47b7-b6d5-248b8a2627ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:44:55.088795 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088726 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f8d78e1-2603-47b7-b6d5-248b8a2627ed" (UID: "2f8d78e1-2603-47b7-b6d5-248b8a2627ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:44:55.088941 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088858 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-model-cache\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:44:55.088941 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088879 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tmp-dir\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:44:55.088941 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.088892 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-home\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:44:55.090763 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.090722 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f8d78e1-2603-47b7-b6d5-248b8a2627ed" (UID: "2f8d78e1-2603-47b7-b6d5-248b8a2627ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 17:44:55.090763 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.090742 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-kube-api-access-ljl6w" (OuterVolumeSpecName: "kube-api-access-ljl6w") pod "2f8d78e1-2603-47b7-b6d5-248b8a2627ed" (UID: "2f8d78e1-2603-47b7-b6d5-248b8a2627ed"). InnerVolumeSpecName "kube-api-access-ljl6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:44:55.090904 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.090788 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-dshm" (OuterVolumeSpecName: "dshm") pod "2f8d78e1-2603-47b7-b6d5-248b8a2627ed" (UID: "2f8d78e1-2603-47b7-b6d5-248b8a2627ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:44:55.190288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.190251 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-tls-certs\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:44:55.190288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.190282 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljl6w\" (UniqueName: \"kubernetes.io/projected/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-kube-api-access-ljl6w\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:44:55.190288 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.190293 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f8d78e1-2603-47b7-b6d5-248b8a2627ed-dshm\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:44:55.914426 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.914399 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" Jul 21 17:44:55.914880 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.914439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4" event={"ID":"2f8d78e1-2603-47b7-b6d5-248b8a2627ed","Type":"ContainerDied","Data":"f43fc1036365d592d395a284b10a8ba575bba3223fd69f7fb39b90d987285c39"} Jul 21 17:44:55.914880 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.914495 2562 scope.go:117] "RemoveContainer" containerID="0c4d5a7d6e76bac53af5351044dfc49409b2c7fead017d8e261194341354e2a3" Jul 21 17:44:55.937077 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.937047 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4"] Jul 21 17:44:55.943890 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:55.943865 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5a36a6/auth-test-model-5a36a6-kserve-76f9cc4879-jvll4"] Jul 21 17:44:56.378148 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:44:56.378090 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8d78e1-2603-47b7-b6d5-248b8a2627ed" path="/var/lib/kubelet/pods/2f8d78e1-2603-47b7-b6d5-248b8a2627ed/volumes" Jul 21 17:45:02.311570 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:02.311533 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:45:02.651693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:02.651602 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:45:02.651693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:02.651656 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:45:02.664013 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:02.663983 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:45:02.961013 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:02.960932 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:45:33.736827 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:33.736742 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:45:37.210571 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:37.210538 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:45:51.612762 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:51.612727 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:45:57.293148 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:45:57.293095 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:11.208962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:11.208923 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:46:17.112592 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:17.112560 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b"] Jul 21 17:46:17.139654 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:17.139615 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:18.055391 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:18.055349 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:20.427258 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.427224 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc"] Jul 21 17:46:20.427854 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.427779 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8d78e1-2603-47b7-b6d5-248b8a2627ed" containerName="main" Jul 21 17:46:20.427854 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.427798 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8d78e1-2603-47b7-b6d5-248b8a2627ed" containerName="main" Jul 21 17:46:20.427979 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.427898 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8d78e1-2603-47b7-b6d5-248b8a2627ed" containerName="main" Jul 21 17:46:20.430908 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.430885 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.433479 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.433460 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6fedc9\"/\"rate-test-model-6fedc9-kserve-self-signed-certs\"" Jul 21 17:46:20.442222 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.442202 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc"] Jul 21 17:46:20.454904 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.454878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:20.458394 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.458369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-model-cache\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.458500 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.458412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmdld\" (UniqueName: \"kubernetes.io/projected/d0bbb80d-84b0-4bfe-be6a-663922603694-kube-api-access-vmdld\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.458500 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.458446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-dshm\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.458636 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.458615 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-home\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.458696 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.458652 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0bbb80d-84b0-4bfe-be6a-663922603694-tls-certs\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.458800 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.458787 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-tmp-dir\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.559982 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.559944 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-model-cache\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.559982 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.559986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmdld\" (UniqueName: \"kubernetes.io/projected/d0bbb80d-84b0-4bfe-be6a-663922603694-kube-api-access-vmdld\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.560289 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.560045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-dshm\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.560289 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.560162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-home\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.560289 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.560186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0bbb80d-84b0-4bfe-be6a-663922603694-tls-certs\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.560289 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.560276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-tmp-dir\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.560616 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.560592 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-model-cache\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.560693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.560611 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-home\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.560693 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.560657 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-tmp-dir\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.562376 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.562358 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-dshm\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.562665 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.562650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0bbb80d-84b0-4bfe-be6a-663922603694-tls-certs\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.568527 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.568505 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmdld\" (UniqueName: \"kubernetes.io/projected/d0bbb80d-84b0-4bfe-be6a-663922603694-kube-api-access-vmdld\") pod \"rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.740260 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.740229 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:20.878344 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:20.877391 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc"] Jul 21 17:46:20.881873 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:46:20.881838 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0bbb80d_84b0_4bfe_be6a_663922603694.slice/crio-0b0be637149584dfbf1daa78540cdf0b0068e79ceb9a98bd9cf2bede3ec2926a WatchSource:0}: Error finding container 0b0be637149584dfbf1daa78540cdf0b0068e79ceb9a98bd9cf2bede3ec2926a: Status 404 returned error can't find the container with id 0b0be637149584dfbf1daa78540cdf0b0068e79ceb9a98bd9cf2bede3ec2926a Jul 21 17:46:21.000279 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:21.000192 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:21.101885 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:21.101849 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:21.306605 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:21.306520 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" event={"ID":"d0bbb80d-84b0-4bfe-be6a-663922603694","Type":"ContainerStarted","Data":"4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75"} Jul 21 17:46:21.306605 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:21.306560 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" event={"ID":"d0bbb80d-84b0-4bfe-be6a-663922603694","Type":"ContainerStarted","Data":"0b0be637149584dfbf1daa78540cdf0b0068e79ceb9a98bd9cf2bede3ec2926a"} Jul 21 17:46:21.324799 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:21.324744 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" podStartSLOduration=1.324725401 podStartE2EDuration="1.324725401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:46:21.324361018 +0000 UTC m=+2649.438508004" watchObservedRunningTime="2026-07-21 17:46:21.324725401 +0000 UTC m=+2649.438872386" Jul 21 17:46:28.016766 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:28.016730 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:46:30.741043 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:30.741001 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:30.741043 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:30.741053 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:30.753511 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:30.753485 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:31.363819 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:31.363790 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:46:32.132750 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.132692 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" podUID="74b40d76-2508-44c0-9f6a-1e5a19202bf4" containerName="main" containerID="cri-o://31b4a4c2bc98f314ddb8118a5b709ce44a84584d6b479bb6bbb71fc2f2f1013f" gracePeriod=45 Jul 21 17:46:32.358539 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.358503 2562 generic.go:356] "Generic (PLEG): container finished" podID="74b40d76-2508-44c0-9f6a-1e5a19202bf4" containerID="31b4a4c2bc98f314ddb8118a5b709ce44a84584d6b479bb6bbb71fc2f2f1013f" exitCode=0 Jul 21 17:46:32.358706 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.358577 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" event={"ID":"74b40d76-2508-44c0-9f6a-1e5a19202bf4","Type":"ContainerDied","Data":"31b4a4c2bc98f314ddb8118a5b709ce44a84584d6b479bb6bbb71fc2f2f1013f"} Jul 21 17:46:32.390747 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.390687 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:46:32.470799 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.470767 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tmp-dir\") pod \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " Jul 21 17:46:32.470799 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.470805 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-home\") pod \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " Jul 21 17:46:32.471020 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.470867 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m5rm\" (UniqueName: \"kubernetes.io/projected/74b40d76-2508-44c0-9f6a-1e5a19202bf4-kube-api-access-6m5rm\") pod \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " Jul 21 17:46:32.471020 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.470912 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-dshm\") pod \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " Jul 21 17:46:32.471020 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.470932 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tls-certs\") pod \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " Jul 21 17:46:32.471020 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.470967 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-model-cache\") pod \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\" (UID: \"74b40d76-2508-44c0-9f6a-1e5a19202bf4\") " Jul 21 17:46:32.471255 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.471065 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "74b40d76-2508-44c0-9f6a-1e5a19202bf4" (UID: "74b40d76-2508-44c0-9f6a-1e5a19202bf4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:46:32.471255 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.471151 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-home" (OuterVolumeSpecName: "home") pod "74b40d76-2508-44c0-9f6a-1e5a19202bf4" (UID: "74b40d76-2508-44c0-9f6a-1e5a19202bf4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:46:32.471364 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.471316 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tmp-dir\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:46:32.471364 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.471334 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-home\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:46:32.471454 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.471372 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-model-cache" (OuterVolumeSpecName: "model-cache") pod "74b40d76-2508-44c0-9f6a-1e5a19202bf4" (UID: "74b40d76-2508-44c0-9f6a-1e5a19202bf4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:46:32.473239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.473203 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74b40d76-2508-44c0-9f6a-1e5a19202bf4" (UID: "74b40d76-2508-44c0-9f6a-1e5a19202bf4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 17:46:32.473543 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.473520 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-dshm" (OuterVolumeSpecName: "dshm") pod "74b40d76-2508-44c0-9f6a-1e5a19202bf4" (UID: "74b40d76-2508-44c0-9f6a-1e5a19202bf4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:46:32.473543 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.473534 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b40d76-2508-44c0-9f6a-1e5a19202bf4-kube-api-access-6m5rm" (OuterVolumeSpecName: "kube-api-access-6m5rm") pod "74b40d76-2508-44c0-9f6a-1e5a19202bf4" (UID: "74b40d76-2508-44c0-9f6a-1e5a19202bf4"). InnerVolumeSpecName "kube-api-access-6m5rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:46:32.572547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.572513 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-dshm\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:46:32.572547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.572541 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74b40d76-2508-44c0-9f6a-1e5a19202bf4-tls-certs\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:46:32.572547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.572551 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74b40d76-2508-44c0-9f6a-1e5a19202bf4-model-cache\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:46:32.572775 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:32.572559 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m5rm\" (UniqueName: \"kubernetes.io/projected/74b40d76-2508-44c0-9f6a-1e5a19202bf4-kube-api-access-6m5rm\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:46:33.364587 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:33.364555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" event={"ID":"74b40d76-2508-44c0-9f6a-1e5a19202bf4","Type":"ContainerDied","Data":"fac2e5ad2496413fca81c77b13e9a76d93465bcf5032bed20196dcd468d3651a"} Jul 21 17:46:33.364587 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:33.364587 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b" Jul 21 17:46:33.364587 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:33.364604 2562 scope.go:117] "RemoveContainer" containerID="31b4a4c2bc98f314ddb8118a5b709ce44a84584d6b479bb6bbb71fc2f2f1013f" Jul 21 17:46:33.386547 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:33.386522 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b"] Jul 21 17:46:33.391733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:33.391702 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/sub-test-model-6fedc9-kserve-6dfb958d85-2b94b"] Jul 21 17:46:34.377590 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:34.377559 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b40d76-2508-44c0-9f6a-1e5a19202bf4" path="/var/lib/kubelet/pods/74b40d76-2508-44c0-9f6a-1e5a19202bf4/volumes" Jul 21 17:46:41.257964 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:41.257925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:44.816165 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:44.816131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:46:50.538148 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:50.538090 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:50.638992 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:50.638946 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:58.694970 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:58.694934 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:46:58.795611 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:46:58.795578 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:47:10.838445 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:10.838372 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:47:29.817632 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:29.817594 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:47:34.007973 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:34.007939 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc"] Jul 21 17:47:34.018859 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:34.018827 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:47:34.735361 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:34.735330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:47:35.685137 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.685087 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-140548771671216"] Jul 21 17:47:35.685701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.685616 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b40d76-2508-44c0-9f6a-1e5a19202bf4" containerName="main" Jul 21 17:47:35.685701 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.685635 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b40d76-2508-44c0-9f6a-1e5a19202bf4" containerName="main" Jul 21 17:47:35.685828 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.685735 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b40d76-2508-44c0-9f6a-1e5a19202bf4" containerName="main" Jul 21 17:47:35.688803 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.688784 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:35.691093 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.691077 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-rfhgq\"" Jul 21 17:47:35.697367 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.697345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771671216"] Jul 21 17:47:35.810424 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.810390 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42gwk\" (UniqueName: \"kubernetes.io/projected/d9d704dd-5d9c-4f45-a663-1ab6797bcc98-kube-api-access-42gwk\") pod \"test-curl-7639-140548771671216\" (UID: \"d9d704dd-5d9c-4f45-a663-1ab6797bcc98\") " pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:35.911899 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.911867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42gwk\" (UniqueName: \"kubernetes.io/projected/d9d704dd-5d9c-4f45-a663-1ab6797bcc98-kube-api-access-42gwk\") pod \"test-curl-7639-140548771671216\" (UID: \"d9d704dd-5d9c-4f45-a663-1ab6797bcc98\") " pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:35.920600 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.920576 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42gwk\" (UniqueName: \"kubernetes.io/projected/d9d704dd-5d9c-4f45-a663-1ab6797bcc98-kube-api-access-42gwk\") pod \"test-curl-7639-140548771671216\" (UID: \"d9d704dd-5d9c-4f45-a663-1ab6797bcc98\") " pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:35.998662 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:35.998635 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:36.128286 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:36.128257 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771671216"] Jul 21 17:47:36.130230 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:47:36.130205 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9d704dd_5d9c_4f45_a663_1ab6797bcc98.slice/crio-68520c1d8939b3badc8a14ac0c47c0001444480730dbd8347764d7cd5d184f47 WatchSource:0}: Error finding container 68520c1d8939b3badc8a14ac0c47c0001444480730dbd8347764d7cd5d184f47: Status 404 returned error can't find the container with id 68520c1d8939b3badc8a14ac0c47c0001444480730dbd8347764d7cd5d184f47 Jul 21 17:47:36.131492 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:36.131465 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 17:47:36.652186 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:36.652138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771671216" event={"ID":"d9d704dd-5d9c-4f45-a663-1ab6797bcc98","Type":"ContainerStarted","Data":"68520c1d8939b3badc8a14ac0c47c0001444480730dbd8347764d7cd5d184f47"} Jul 21 17:47:37.664334 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:37.664295 2562 generic.go:356] "Generic (PLEG): container finished" podID="d9d704dd-5d9c-4f45-a663-1ab6797bcc98" containerID="0590d0e174aab16803f866bf2046c51492f19fbeb4ed14603ff84f86a1c8f460" exitCode=0 Jul 21 17:47:37.664728 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:37.664381 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771671216" event={"ID":"d9d704dd-5d9c-4f45-a663-1ab6797bcc98","Type":"ContainerDied","Data":"0590d0e174aab16803f866bf2046c51492f19fbeb4ed14603ff84f86a1c8f460"} Jul 21 17:47:38.801865 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.801838 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:38.823945 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.823916 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7639-140548771671216_d9d704dd-5d9c-4f45-a663-1ab6797bcc98/test-curl-7639-140548771671216/0.log" Jul 21 17:47:38.849866 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.849833 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-140548771671216"] Jul 21 17:47:38.854085 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.854060 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-140548771671216"] Jul 21 17:47:38.937301 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.937231 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42gwk\" (UniqueName: \"kubernetes.io/projected/d9d704dd-5d9c-4f45-a663-1ab6797bcc98-kube-api-access-42gwk\") pod \"d9d704dd-5d9c-4f45-a663-1ab6797bcc98\" (UID: \"d9d704dd-5d9c-4f45-a663-1ab6797bcc98\") " Jul 21 17:47:38.939417 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.939390 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d704dd-5d9c-4f45-a663-1ab6797bcc98-kube-api-access-42gwk" (OuterVolumeSpecName: "kube-api-access-42gwk") pod "d9d704dd-5d9c-4f45-a663-1ab6797bcc98" (UID: "d9d704dd-5d9c-4f45-a663-1ab6797bcc98"). InnerVolumeSpecName "kube-api-access-42gwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:47:38.986092 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.986054 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-140548771669808"] Jul 21 17:47:38.986497 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.986483 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9d704dd-5d9c-4f45-a663-1ab6797bcc98" containerName="test-curl-7639-140548771671216" Jul 21 17:47:38.986545 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.986499 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9d704dd-5d9c-4f45-a663-1ab6797bcc98" containerName="test-curl-7639-140548771671216" Jul 21 17:47:38.986577 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.986572 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9d704dd-5d9c-4f45-a663-1ab6797bcc98" containerName="test-curl-7639-140548771671216" Jul 21 17:47:38.989666 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.989651 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771669808" Jul 21 17:47:38.994901 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:38.994873 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771669808"] Jul 21 17:47:39.038376 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.038337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7lzx\" (UniqueName: \"kubernetes.io/projected/b009f22f-e122-461c-aca1-c6f135430f77-kube-api-access-j7lzx\") pod \"test-curl-7639-140548771669808\" (UID: \"b009f22f-e122-461c-aca1-c6f135430f77\") " pod="opendatahub/test-curl-7639-140548771669808" Jul 21 17:47:39.038544 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.038495 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42gwk\" (UniqueName: \"kubernetes.io/projected/d9d704dd-5d9c-4f45-a663-1ab6797bcc98-kube-api-access-42gwk\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:39.139848 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.139804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7lzx\" (UniqueName: \"kubernetes.io/projected/b009f22f-e122-461c-aca1-c6f135430f77-kube-api-access-j7lzx\") pod \"test-curl-7639-140548771669808\" (UID: \"b009f22f-e122-461c-aca1-c6f135430f77\") " pod="opendatahub/test-curl-7639-140548771669808" Jul 21 17:47:39.149216 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.149189 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7lzx\" (UniqueName: \"kubernetes.io/projected/b009f22f-e122-461c-aca1-c6f135430f77-kube-api-access-j7lzx\") pod \"test-curl-7639-140548771669808\" (UID: \"b009f22f-e122-461c-aca1-c6f135430f77\") " pod="opendatahub/test-curl-7639-140548771669808" Jul 21 17:47:39.299667 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.299620 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771669808" Jul 21 17:47:39.429123 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.429085 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771669808"] Jul 21 17:47:39.430933 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:47:39.430907 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb009f22f_e122_461c_aca1_c6f135430f77.slice/crio-ec8b52ccfbde6d550a63263beac72ef67ef0c3adebeb8652f9464e3f6c101451 WatchSource:0}: Error finding container ec8b52ccfbde6d550a63263beac72ef67ef0c3adebeb8652f9464e3f6c101451: Status 404 returned error can't find the container with id ec8b52ccfbde6d550a63263beac72ef67ef0c3adebeb8652f9464e3f6c101451 Jul 21 17:47:39.673885 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.673791 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771669808" event={"ID":"b009f22f-e122-461c-aca1-c6f135430f77","Type":"ContainerStarted","Data":"ec8b52ccfbde6d550a63263beac72ef67ef0c3adebeb8652f9464e3f6c101451"} Jul 21 17:47:39.674940 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.674922 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:39.674940 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:39.674937 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68520c1d8939b3badc8a14ac0c47c0001444480730dbd8347764d7cd5d184f47" Jul 21 17:47:39.685248 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:47:39.685223 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7639-140548771671216\" is forbidden: User \"system:node:ip-10-0-134-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-97.ec2.internal' and this object" podUID="d9d704dd-5d9c-4f45-a663-1ab6797bcc98" pod="opendatahub/test-curl-7639-140548771671216" Jul 21 17:47:40.378779 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:40.378745 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9d704dd-5d9c-4f45-a663-1ab6797bcc98" path="/var/lib/kubelet/pods/d9d704dd-5d9c-4f45-a663-1ab6797bcc98/volumes" Jul 21 17:47:40.680766 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:40.680689 2562 generic.go:356] "Generic (PLEG): container finished" podID="b009f22f-e122-461c-aca1-c6f135430f77" containerID="c154cdcd97005fb4c430d6c1b36fa13a6896c3e367250ed5358d12712679d45e" exitCode=0 Jul 21 17:47:40.680902 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:40.680766 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771669808" event={"ID":"b009f22f-e122-461c-aca1-c6f135430f77","Type":"ContainerDied","Data":"c154cdcd97005fb4c430d6c1b36fa13a6896c3e367250ed5358d12712679d45e"} Jul 21 17:47:41.817086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:41.817062 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771669808" Jul 21 17:47:41.838862 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:41.838824 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7639-140548771669808_b009f22f-e122-461c-aca1-c6f135430f77/test-curl-7639-140548771669808/0.log" Jul 21 17:47:41.864365 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:41.864332 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-140548771669808"] Jul 21 17:47:41.866767 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:41.866744 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-140548771669808"] Jul 21 17:47:41.969334 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:41.969303 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7lzx\" (UniqueName: \"kubernetes.io/projected/b009f22f-e122-461c-aca1-c6f135430f77-kube-api-access-j7lzx\") pod \"b009f22f-e122-461c-aca1-c6f135430f77\" (UID: \"b009f22f-e122-461c-aca1-c6f135430f77\") " Jul 21 17:47:41.971364 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:41.971341 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b009f22f-e122-461c-aca1-c6f135430f77-kube-api-access-j7lzx" (OuterVolumeSpecName: "kube-api-access-j7lzx") pod "b009f22f-e122-461c-aca1-c6f135430f77" (UID: "b009f22f-e122-461c-aca1-c6f135430f77"). InnerVolumeSpecName "kube-api-access-j7lzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:47:42.022931 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.022899 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:42.023325 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.023309 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b009f22f-e122-461c-aca1-c6f135430f77" containerName="test-curl-7639-140548771669808" Jul 21 17:47:42.023390 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.023327 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b009f22f-e122-461c-aca1-c6f135430f77" containerName="test-curl-7639-140548771669808" Jul 21 17:47:42.023430 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.023393 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b009f22f-e122-461c-aca1-c6f135430f77" containerName="test-curl-7639-140548771669808" Jul 21 17:47:42.026352 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.026333 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:42.034219 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.034187 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:42.070045 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.070020 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7lzx\" (UniqueName: \"kubernetes.io/projected/b009f22f-e122-461c-aca1-c6f135430f77-kube-api-access-j7lzx\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:42.170753 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.170718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvjh4\" (UniqueName: \"kubernetes.io/projected/135df2a7-7655-4e0d-8f27-ddcca179eae7-kube-api-access-kvjh4\") pod \"test-curl-7639-140548771607584\" (UID: \"135df2a7-7655-4e0d-8f27-ddcca179eae7\") " pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:42.271647 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.271541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvjh4\" (UniqueName: \"kubernetes.io/projected/135df2a7-7655-4e0d-8f27-ddcca179eae7-kube-api-access-kvjh4\") pod \"test-curl-7639-140548771607584\" (UID: \"135df2a7-7655-4e0d-8f27-ddcca179eae7\") " pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:42.280483 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.280451 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvjh4\" (UniqueName: \"kubernetes.io/projected/135df2a7-7655-4e0d-8f27-ddcca179eae7-kube-api-access-kvjh4\") pod \"test-curl-7639-140548771607584\" (UID: \"135df2a7-7655-4e0d-8f27-ddcca179eae7\") " pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:42.336219 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.336186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:42.379964 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.379932 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b009f22f-e122-461c-aca1-c6f135430f77" path="/var/lib/kubelet/pods/b009f22f-e122-461c-aca1-c6f135430f77/volumes" Jul 21 17:47:42.469220 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.469196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:42.471066 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:47:42.471044 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod135df2a7_7655_4e0d_8f27_ddcca179eae7.slice/crio-89984d9c0a5461ce4969bebd9860061ab52b103e2e7d8cc35d1e78d875798eb7 WatchSource:0}: Error finding container 89984d9c0a5461ce4969bebd9860061ab52b103e2e7d8cc35d1e78d875798eb7: Status 404 returned error can't find the container with id 89984d9c0a5461ce4969bebd9860061ab52b103e2e7d8cc35d1e78d875798eb7 Jul 21 17:47:42.691088 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.690999 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771669808" Jul 21 17:47:42.691088 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.691046 2562 scope.go:117] "RemoveContainer" containerID="c154cdcd97005fb4c430d6c1b36fa13a6896c3e367250ed5358d12712679d45e" Jul 21 17:47:42.692406 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:42.692290 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771607584" event={"ID":"135df2a7-7655-4e0d-8f27-ddcca179eae7","Type":"ContainerStarted","Data":"89984d9c0a5461ce4969bebd9860061ab52b103e2e7d8cc35d1e78d875798eb7"} Jul 21 17:47:43.698255 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:43.698220 2562 generic.go:356] "Generic (PLEG): container finished" podID="135df2a7-7655-4e0d-8f27-ddcca179eae7" containerID="d5113838f7f79a9b3b3345a460f35225b4eb7f6f142b7eb7b2629808b137016d" exitCode=0 Jul 21 17:47:43.698662 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:43.698310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771607584" event={"ID":"135df2a7-7655-4e0d-8f27-ddcca179eae7","Type":"ContainerDied","Data":"d5113838f7f79a9b3b3345a460f35225b4eb7f6f142b7eb7b2629808b137016d"} Jul 21 17:47:44.837230 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:44.837205 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:44.859253 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:44.859215 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7639-140548771607584_135df2a7-7655-4e0d-8f27-ddcca179eae7/test-curl-7639-140548771607584/0.log" Jul 21 17:47:44.883424 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:44.883385 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:44.887869 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:44.887841 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:44.995649 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:44.995622 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvjh4\" (UniqueName: \"kubernetes.io/projected/135df2a7-7655-4e0d-8f27-ddcca179eae7-kube-api-access-kvjh4\") pod \"135df2a7-7655-4e0d-8f27-ddcca179eae7\" (UID: \"135df2a7-7655-4e0d-8f27-ddcca179eae7\") " Jul 21 17:47:44.997630 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:44.997604 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135df2a7-7655-4e0d-8f27-ddcca179eae7-kube-api-access-kvjh4" (OuterVolumeSpecName: "kube-api-access-kvjh4") pod "135df2a7-7655-4e0d-8f27-ddcca179eae7" (UID: "135df2a7-7655-4e0d-8f27-ddcca179eae7"). InnerVolumeSpecName "kube-api-access-kvjh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:47:45.005306 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.005286 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:45.005653 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.005637 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="135df2a7-7655-4e0d-8f27-ddcca179eae7" containerName="test-curl-7639-140548771607584" Jul 21 17:47:45.005653 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.005651 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="135df2a7-7655-4e0d-8f27-ddcca179eae7" containerName="test-curl-7639-140548771607584" Jul 21 17:47:45.005801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.005721 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="135df2a7-7655-4e0d-8f27-ddcca179eae7" containerName="test-curl-7639-140548771607584" Jul 21 17:47:45.008634 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.008618 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:45.016361 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.016342 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:45.096394 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.096361 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfbd\" (UniqueName: \"kubernetes.io/projected/fab60cad-6d00-4b7e-9487-25635a5a9480-kube-api-access-wkfbd\") pod \"test-curl-7639-140548771607584\" (UID: \"fab60cad-6d00-4b7e-9487-25635a5a9480\") " pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:45.096558 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.096517 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvjh4\" (UniqueName: \"kubernetes.io/projected/135df2a7-7655-4e0d-8f27-ddcca179eae7-kube-api-access-kvjh4\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:45.197250 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.197209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfbd\" (UniqueName: \"kubernetes.io/projected/fab60cad-6d00-4b7e-9487-25635a5a9480-kube-api-access-wkfbd\") pod \"test-curl-7639-140548771607584\" (UID: \"fab60cad-6d00-4b7e-9487-25635a5a9480\") " pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:45.207325 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.207295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfbd\" (UniqueName: \"kubernetes.io/projected/fab60cad-6d00-4b7e-9487-25635a5a9480-kube-api-access-wkfbd\") pod \"test-curl-7639-140548771607584\" (UID: \"fab60cad-6d00-4b7e-9487-25635a5a9480\") " pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:45.318795 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.318707 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:45.450792 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.450770 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:45.453047 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:47:45.453022 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab60cad_6d00_4b7e_9487_25635a5a9480.slice/crio-d15a82eac981e12e1a7ff0f9bac46d46f198fdda83b1dcc97a305f319a0ed4a6 WatchSource:0}: Error finding container d15a82eac981e12e1a7ff0f9bac46d46f198fdda83b1dcc97a305f319a0ed4a6: Status 404 returned error can't find the container with id d15a82eac981e12e1a7ff0f9bac46d46f198fdda83b1dcc97a305f319a0ed4a6 Jul 21 17:47:45.708607 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.708533 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89984d9c0a5461ce4969bebd9860061ab52b103e2e7d8cc35d1e78d875798eb7" Jul 21 17:47:45.708607 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.708531 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:45.709691 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.709666 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771607584" event={"ID":"fab60cad-6d00-4b7e-9487-25635a5a9480","Type":"ContainerStarted","Data":"d15a82eac981e12e1a7ff0f9bac46d46f198fdda83b1dcc97a305f319a0ed4a6"} Jul 21 17:47:45.720455 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:45.720425 2562 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7639-140548771607584" oldPodUID="135df2a7-7655-4e0d-8f27-ddcca179eae7" podUID="fab60cad-6d00-4b7e-9487-25635a5a9480" Jul 21 17:47:46.377848 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:46.377808 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="135df2a7-7655-4e0d-8f27-ddcca179eae7" path="/var/lib/kubelet/pods/135df2a7-7655-4e0d-8f27-ddcca179eae7/volumes" Jul 21 17:47:46.517970 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:46.517937 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:47:46.716665 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:46.716574 2562 generic.go:356] "Generic (PLEG): container finished" podID="fab60cad-6d00-4b7e-9487-25635a5a9480" containerID="29ebbd9df9b1bfc9ee67a6d2e6d26a056debac25ee7188e04e6b1d4620ba7cff" exitCode=0 Jul 21 17:47:46.716665 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:46.716626 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771607584" event={"ID":"fab60cad-6d00-4b7e-9487-25635a5a9480","Type":"ContainerDied","Data":"29ebbd9df9b1bfc9ee67a6d2e6d26a056debac25ee7188e04e6b1d4620ba7cff"} Jul 21 17:47:47.853699 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:47.853675 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:47.876084 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:47.876050 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7639-140548771607584_fab60cad-6d00-4b7e-9487-25635a5a9480/test-curl-7639-140548771607584/0.log" Jul 21 17:47:47.901183 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:47.901156 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:47.906397 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:47.906377 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-140548771607584"] Jul 21 17:47:47.920041 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:47.920019 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkfbd\" (UniqueName: \"kubernetes.io/projected/fab60cad-6d00-4b7e-9487-25635a5a9480-kube-api-access-wkfbd\") pod \"fab60cad-6d00-4b7e-9487-25635a5a9480\" (UID: \"fab60cad-6d00-4b7e-9487-25635a5a9480\") " Jul 21 17:47:47.921969 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:47.921942 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab60cad-6d00-4b7e-9487-25635a5a9480-kube-api-access-wkfbd" (OuterVolumeSpecName: "kube-api-access-wkfbd") pod "fab60cad-6d00-4b7e-9487-25635a5a9480" (UID: "fab60cad-6d00-4b7e-9487-25635a5a9480"). InnerVolumeSpecName "kube-api-access-wkfbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:47:48.021022 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.020952 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7639-140548771608592"] Jul 21 17:47:48.021286 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.021268 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkfbd\" (UniqueName: \"kubernetes.io/projected/fab60cad-6d00-4b7e-9487-25635a5a9480-kube-api-access-wkfbd\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:48.021376 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.021362 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab60cad-6d00-4b7e-9487-25635a5a9480" containerName="test-curl-7639-140548771607584" Jul 21 17:47:48.021414 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.021379 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab60cad-6d00-4b7e-9487-25635a5a9480" containerName="test-curl-7639-140548771607584" Jul 21 17:47:48.021451 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.021434 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab60cad-6d00-4b7e-9487-25635a5a9480" containerName="test-curl-7639-140548771607584" Jul 21 17:47:48.024309 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.024295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:48.032058 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.032035 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771608592"] Jul 21 17:47:48.122327 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.122280 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97cd\" (UniqueName: \"kubernetes.io/projected/d0aa8536-d2c6-41dd-9cf8-3a4081749dc5-kube-api-access-r97cd\") pod \"test-curl-7639-140548771608592\" (UID: \"d0aa8536-d2c6-41dd-9cf8-3a4081749dc5\") " pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:48.223515 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.223474 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97cd\" (UniqueName: \"kubernetes.io/projected/d0aa8536-d2c6-41dd-9cf8-3a4081749dc5-kube-api-access-r97cd\") pod \"test-curl-7639-140548771608592\" (UID: \"d0aa8536-d2c6-41dd-9cf8-3a4081749dc5\") " pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:48.233126 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.233068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97cd\" (UniqueName: \"kubernetes.io/projected/d0aa8536-d2c6-41dd-9cf8-3a4081749dc5-kube-api-access-r97cd\") pod \"test-curl-7639-140548771608592\" (UID: \"d0aa8536-d2c6-41dd-9cf8-3a4081749dc5\") " pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:48.334225 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.334128 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:48.378562 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.378524 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab60cad-6d00-4b7e-9487-25635a5a9480" path="/var/lib/kubelet/pods/fab60cad-6d00-4b7e-9487-25635a5a9480/volumes" Jul 21 17:47:48.466518 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.466485 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7639-140548771608592"] Jul 21 17:47:48.468956 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:47:48.468925 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0aa8536_d2c6_41dd_9cf8_3a4081749dc5.slice/crio-c9dd3183b7e59ddbe1ee58ebbb134660c35608091183884cdecad21ea88c54b6 WatchSource:0}: Error finding container c9dd3183b7e59ddbe1ee58ebbb134660c35608091183884cdecad21ea88c54b6: Status 404 returned error can't find the container with id c9dd3183b7e59ddbe1ee58ebbb134660c35608091183884cdecad21ea88c54b6 Jul 21 17:47:48.727834 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.727793 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771608592" event={"ID":"d0aa8536-d2c6-41dd-9cf8-3a4081749dc5","Type":"ContainerStarted","Data":"c9dd3183b7e59ddbe1ee58ebbb134660c35608091183884cdecad21ea88c54b6"} Jul 21 17:47:48.728925 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.728903 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771607584" Jul 21 17:47:48.728925 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:48.728919 2562 scope.go:117] "RemoveContainer" containerID="29ebbd9df9b1bfc9ee67a6d2e6d26a056debac25ee7188e04e6b1d4620ba7cff" Jul 21 17:47:49.027995 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.027908 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" podUID="d0bbb80d-84b0-4bfe-be6a-663922603694" containerName="main" containerID="cri-o://4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75" gracePeriod=45 Jul 21 17:47:49.276442 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.276415 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:47:49.433715 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.433628 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmdld\" (UniqueName: \"kubernetes.io/projected/d0bbb80d-84b0-4bfe-be6a-663922603694-kube-api-access-vmdld\") pod \"d0bbb80d-84b0-4bfe-be6a-663922603694\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " Jul 21 17:47:49.433715 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.433678 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-model-cache\") pod \"d0bbb80d-84b0-4bfe-be6a-663922603694\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " Jul 21 17:47:49.433913 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.433720 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-dshm\") pod \"d0bbb80d-84b0-4bfe-be6a-663922603694\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " Jul 21 17:47:49.433913 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.433749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-home\") pod \"d0bbb80d-84b0-4bfe-be6a-663922603694\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " Jul 21 17:47:49.433913 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.433775 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0bbb80d-84b0-4bfe-be6a-663922603694-tls-certs\") pod \"d0bbb80d-84b0-4bfe-be6a-663922603694\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " Jul 21 17:47:49.433913 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.433843 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-tmp-dir\") pod \"d0bbb80d-84b0-4bfe-be6a-663922603694\" (UID: \"d0bbb80d-84b0-4bfe-be6a-663922603694\") " Jul 21 17:47:49.434139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.433997 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0bbb80d-84b0-4bfe-be6a-663922603694" (UID: "d0bbb80d-84b0-4bfe-be6a-663922603694"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:47:49.434139 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.434020 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-home" (OuterVolumeSpecName: "home") pod "d0bbb80d-84b0-4bfe-be6a-663922603694" (UID: "d0bbb80d-84b0-4bfe-be6a-663922603694"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:47:49.434257 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.434152 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-model-cache\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:49.434257 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.434165 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-home\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:49.434257 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.434166 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0bbb80d-84b0-4bfe-be6a-663922603694" (UID: "d0bbb80d-84b0-4bfe-be6a-663922603694"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:47:49.435806 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.435784 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-dshm" (OuterVolumeSpecName: "dshm") pod "d0bbb80d-84b0-4bfe-be6a-663922603694" (UID: "d0bbb80d-84b0-4bfe-be6a-663922603694"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 17:47:49.435806 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.435794 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bbb80d-84b0-4bfe-be6a-663922603694-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0bbb80d-84b0-4bfe-be6a-663922603694" (UID: "d0bbb80d-84b0-4bfe-be6a-663922603694"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 17:47:49.435924 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.435860 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bbb80d-84b0-4bfe-be6a-663922603694-kube-api-access-vmdld" (OuterVolumeSpecName: "kube-api-access-vmdld") pod "d0bbb80d-84b0-4bfe-be6a-663922603694" (UID: "d0bbb80d-84b0-4bfe-be6a-663922603694"). InnerVolumeSpecName "kube-api-access-vmdld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:47:49.534980 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.534942 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0bbb80d-84b0-4bfe-be6a-663922603694-tls-certs\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:49.534980 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.534972 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-tmp-dir\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:49.534980 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.534987 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmdld\" (UniqueName: \"kubernetes.io/projected/d0bbb80d-84b0-4bfe-be6a-663922603694-kube-api-access-vmdld\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:49.535221 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.534998 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0bbb80d-84b0-4bfe-be6a-663922603694-dshm\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:49.735576 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.735544 2562 generic.go:356] "Generic (PLEG): container finished" podID="d0bbb80d-84b0-4bfe-be6a-663922603694" containerID="4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75" exitCode=0 Jul 21 17:47:49.735765 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.735590 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" Jul 21 17:47:49.735765 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.735615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" event={"ID":"d0bbb80d-84b0-4bfe-be6a-663922603694","Type":"ContainerDied","Data":"4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75"} Jul 21 17:47:49.735765 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.735662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc" event={"ID":"d0bbb80d-84b0-4bfe-be6a-663922603694","Type":"ContainerDied","Data":"0b0be637149584dfbf1daa78540cdf0b0068e79ceb9a98bd9cf2bede3ec2926a"} Jul 21 17:47:49.735765 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.735691 2562 scope.go:117] "RemoveContainer" containerID="4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75" Jul 21 17:47:49.737420 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.737397 2562 generic.go:356] "Generic (PLEG): container finished" podID="d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" containerID="5c9e3b029b71c645fb00303ded42b2a97455cd811bb0dda0eaea26e3e9a81637" exitCode=0 Jul 21 17:47:49.737539 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.737481 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7639-140548771608592" event={"ID":"d0aa8536-d2c6-41dd-9cf8-3a4081749dc5","Type":"ContainerDied","Data":"5c9e3b029b71c645fb00303ded42b2a97455cd811bb0dda0eaea26e3e9a81637"} Jul 21 17:47:49.744999 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.744878 2562 scope.go:117] "RemoveContainer" containerID="4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75" Jul 21 17:47:49.745190 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:47:49.745164 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75\": container with ID starting with 4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75 not found: ID does not exist" containerID="4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75" Jul 21 17:47:49.745318 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.745202 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75"} err="failed to get container status \"4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75\": rpc error: code = NotFound desc = could not find container \"4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75\": container with ID starting with 4f06ab49b405cf259fd7402d4672b248b69f4e672df6e87b03b1d4d62a02ab75 not found: ID does not exist" Jul 21 17:47:49.769506 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.769472 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc"] Jul 21 17:47:49.774466 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:49.774441 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6fedc9/rate-test-model-6fedc9-kserve-5ddd56df9f-znmtc"] Jul 21 17:47:50.377918 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:50.377886 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0bbb80d-84b0-4bfe-be6a-663922603694" path="/var/lib/kubelet/pods/d0bbb80d-84b0-4bfe-be6a-663922603694/volumes" Jul 21 17:47:50.896136 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:50.896095 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:50.916974 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:50.916945 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7639-140548771608592_d0aa8536-d2c6-41dd-9cf8-3a4081749dc5/test-curl-7639-140548771608592/0.log" Jul 21 17:47:50.941722 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:50.941698 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7639-140548771608592"] Jul 21 17:47:50.943971 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:50.943949 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7639-140548771608592"] Jul 21 17:47:51.047956 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:51.047861 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r97cd\" (UniqueName: \"kubernetes.io/projected/d0aa8536-d2c6-41dd-9cf8-3a4081749dc5-kube-api-access-r97cd\") pod \"d0aa8536-d2c6-41dd-9cf8-3a4081749dc5\" (UID: \"d0aa8536-d2c6-41dd-9cf8-3a4081749dc5\") " Jul 21 17:47:51.049890 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:51.049861 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0aa8536-d2c6-41dd-9cf8-3a4081749dc5-kube-api-access-r97cd" (OuterVolumeSpecName: "kube-api-access-r97cd") pod "d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" (UID: "d0aa8536-d2c6-41dd-9cf8-3a4081749dc5"). InnerVolumeSpecName "kube-api-access-r97cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 17:47:51.149504 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:51.149468 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r97cd\" (UniqueName: \"kubernetes.io/projected/d0aa8536-d2c6-41dd-9cf8-3a4081749dc5-kube-api-access-r97cd\") on node \"ip-10-0-134-97.ec2.internal\" DevicePath \"\"" Jul 21 17:47:51.752951 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:51.752925 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9dd3183b7e59ddbe1ee58ebbb134660c35608091183884cdecad21ea88c54b6" Jul 21 17:47:51.752951 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:51.752954 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:51.762376 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:47:51.762349 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7639-140548771608592\" is forbidden: User \"system:node:ip-10-0-134-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-97.ec2.internal' and this object" podUID="d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:52.381748 ip-10-0-134-97 kubenswrapper[2562]: E0721 17:47:52.381707 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7639-140548771608592\" is forbidden: User \"system:node:ip-10-0-134-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-97.ec2.internal' and this object" podUID="d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" pod="opendatahub/test-curl-7639-140548771608592" Jul 21 17:47:52.383015 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:52.382995 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" path="/var/lib/kubelet/pods/d0aa8536-d2c6-41dd-9cf8-3a4081749dc5/volumes" Jul 21 17:47:55.224440 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:55.224403 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:47:55.400985 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:47:55.400941 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:04.934276 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:04.934241 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:07.138345 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:07.138314 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:08.213074 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:08.213036 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:48:09.556868 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:09.556826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:09.934777 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:09.934694 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:16.211239 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:16.211199 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:16.311672 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:16.311641 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:16.487115 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:16.487067 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:16.702570 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:16.702537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:17.627033 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:17.626997 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:17.729535 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:17.729502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:27.328180 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:27.328133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:30.015033 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:30.014999 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8q8h7"] Jul 21 17:48:30.625610 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:30.625558 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:30.685839 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:30.685801 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:31.917853 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:31.917822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:32.316407 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:32.316374 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:38.888870 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:38.888781 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:38.989799 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:38.989761 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:39.172035 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:39.171955 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:39.398622 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:39.398588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:39.830727 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:39.830693 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584899858f-5f22d_157ed407-95c2-4154-a862-4fd938b2bf2b/authorino/0.log" Jul 21 17:48:45.395888 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:45.395851 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8vldc_79fcc563-9de5-404b-8bc4-f8763ebdf35f/manager/0.log" Jul 21 17:48:45.627113 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:45.627085 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-npvmb_7102ce76-14d8-4e50-af46-33c80a7c8ac9/server/0.log" Jul 21 17:48:45.995603 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:45.995574 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk_fa65b591-bb96-4c3b-be95-cfd71f969bee/copy-manifests/0.log" Jul 21 17:48:46.014592 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:46.014563 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk_fa65b591-bb96-4c3b-be95-cfd71f969bee/manager/0.log" Jul 21 17:48:48.197640 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:48.197610 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584899858f-5f22d_157ed407-95c2-4154-a862-4fd938b2bf2b/authorino/0.log" Jul 21 17:48:48.547491 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:48.547468 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bsdhw_623f7981-b0d0-4897-afd1-1d822b56e8ac/kuadrant-console-plugin/0.log" Jul 21 17:48:48.888781 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:48.888757 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8q8h7_0a3f56fa-75cd-47cc-afd6-99e9aae217c9/limitador/0.log" Jul 21 17:48:49.693263 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:49.693231 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66868cbd5-s6nbp_d486d602-ccaf-4538-bcf1-8b0a93a35fde/kube-auth-proxy/0.log" Jul 21 17:48:50.833133 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:50.833083 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2nn8p_ca8f4c33-04a2-4365-abde-33a06247e05d/main/0.log" Jul 21 17:48:51.056704 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:51.056671 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-wjh4z_3c102820-9a1d-4ea1-b1f8-4958c4c0e3f7/main/0.log" Jul 21 17:48:56.984512 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:56.984470 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:57.070086 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:57.070052 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" limit=1024 Jul 21 17:48:59.900956 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:48:59.900927 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6xnc7_4835f1ed-ad53-45e6-b120-b08aa18e8baf/global-pull-secret-syncer/0.log" Jul 21 17:49:00.050563 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:00.050526 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qwbq2_3a6b668f-679a-4dc5-9fae-2e3cf8c109d4/konnectivity-agent/0.log" Jul 21 17:49:00.149138 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:00.149090 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-97.ec2.internal_4d8c58923675c06522e424156ee02f34/haproxy/0.log" Jul 21 17:49:05.371520 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:05.371474 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-584899858f-5f22d_157ed407-95c2-4154-a862-4fd938b2bf2b/authorino/0.log" Jul 21 17:49:05.467326 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:05.467297 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bsdhw_623f7981-b0d0-4897-afd1-1d822b56e8ac/kuadrant-console-plugin/0.log" Jul 21 17:49:05.770274 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:05.770245 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8q8h7_0a3f56fa-75cd-47cc-afd6-99e9aae217c9/limitador/0.log" Jul 21 17:49:07.522303 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.522275 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a84dd8ff-b113-4c48-bea5-0382fa878cb3/alertmanager/0.log" Jul 21 17:49:07.556069 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.556041 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a84dd8ff-b113-4c48-bea5-0382fa878cb3/config-reloader/0.log" Jul 21 17:49:07.586973 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.586947 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a84dd8ff-b113-4c48-bea5-0382fa878cb3/kube-rbac-proxy-web/0.log" Jul 21 17:49:07.614806 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.614777 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a84dd8ff-b113-4c48-bea5-0382fa878cb3/kube-rbac-proxy/0.log" Jul 21 17:49:07.644092 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.644065 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a84dd8ff-b113-4c48-bea5-0382fa878cb3/kube-rbac-proxy-metric/0.log" Jul 21 17:49:07.671801 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.671772 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a84dd8ff-b113-4c48-bea5-0382fa878cb3/prom-label-proxy/0.log" Jul 21 17:49:07.703628 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.703605 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a84dd8ff-b113-4c48-bea5-0382fa878cb3/init-config-reloader/0.log" Jul 21 17:49:07.913792 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:07.913752 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-vvnz9_5ec0f1e7-2f84-49e2-a209-2c5cd4a318f0/monitoring-plugin/0.log" Jul 21 17:49:08.153714 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.153632 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhlsg_82e0c873-81d6-4d5c-8ebd-1bb8d245d808/node-exporter/0.log" Jul 21 17:49:08.178607 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.178586 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhlsg_82e0c873-81d6-4d5c-8ebd-1bb8d245d808/kube-rbac-proxy/0.log" Jul 21 17:49:08.205391 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.205370 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bhlsg_82e0c873-81d6-4d5c-8ebd-1bb8d245d808/init-textfile/0.log" Jul 21 17:49:08.234431 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.234407 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-z4bnh_acfb9a1c-0cbd-4236-9e4d-ec04876b8c18/kube-rbac-proxy-main/0.log" Jul 21 17:49:08.261816 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.261790 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-z4bnh_acfb9a1c-0cbd-4236-9e4d-ec04876b8c18/kube-rbac-proxy-self/0.log" Jul 21 17:49:08.287733 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.287710 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-z4bnh_acfb9a1c-0cbd-4236-9e4d-ec04876b8c18/openshift-state-metrics/0.log" Jul 21 17:49:08.325559 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.325533 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_764930c1-c169-4bc9-8c14-cd3b7a5e5659/prometheus/0.log" Jul 21 17:49:08.349340 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.349320 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_764930c1-c169-4bc9-8c14-cd3b7a5e5659/config-reloader/0.log" Jul 21 17:49:08.384116 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.384070 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_764930c1-c169-4bc9-8c14-cd3b7a5e5659/thanos-sidecar/0.log" Jul 21 17:49:08.404998 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.404920 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_764930c1-c169-4bc9-8c14-cd3b7a5e5659/kube-rbac-proxy-web/0.log" Jul 21 17:49:08.432697 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.432664 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_764930c1-c169-4bc9-8c14-cd3b7a5e5659/kube-rbac-proxy/0.log" Jul 21 17:49:08.458195 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.458172 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_764930c1-c169-4bc9-8c14-cd3b7a5e5659/kube-rbac-proxy-thanos/0.log" Jul 21 17:49:08.485244 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.485221 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_764930c1-c169-4bc9-8c14-cd3b7a5e5659/init-config-reloader/0.log" Jul 21 17:49:08.519962 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.519936 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-lqlgj_7c55c061-693b-43bf-9cc3-e469aa0eccf5/prometheus-operator/0.log" Jul 21 17:49:08.544887 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.544855 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-lqlgj_7c55c061-693b-43bf-9cc3-e469aa0eccf5/kube-rbac-proxy/0.log" Jul 21 17:49:08.695809 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.695721 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bc9b87b4-j2s65_9f737a90-b272-41e2-b5a4-4c6b3be38f59/thanos-query/0.log" Jul 21 17:49:08.724237 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.724207 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bc9b87b4-j2s65_9f737a90-b272-41e2-b5a4-4c6b3be38f59/kube-rbac-proxy-web/0.log" Jul 21 17:49:08.748339 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.748309 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bc9b87b4-j2s65_9f737a90-b272-41e2-b5a4-4c6b3be38f59/kube-rbac-proxy/0.log" Jul 21 17:49:08.774063 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.774045 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bc9b87b4-j2s65_9f737a90-b272-41e2-b5a4-4c6b3be38f59/prom-label-proxy/0.log" Jul 21 17:49:08.799451 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.799428 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bc9b87b4-j2s65_9f737a90-b272-41e2-b5a4-4c6b3be38f59/kube-rbac-proxy-rules/0.log" Jul 21 17:49:08.828690 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:08.828670 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-bc9b87b4-j2s65_9f737a90-b272-41e2-b5a4-4c6b3be38f59/kube-rbac-proxy-metrics/0.log" Jul 21 17:49:09.994536 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.994505 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb"] Jul 21 17:49:09.994946 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.994878 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bbb80d-84b0-4bfe-be6a-663922603694" containerName="main" Jul 21 17:49:09.994946 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.994890 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bbb80d-84b0-4bfe-be6a-663922603694" containerName="main" Jul 21 17:49:09.994946 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.994899 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" containerName="test-curl-7639-140548771608592" Jul 21 17:49:09.994946 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.994905 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" containerName="test-curl-7639-140548771608592" Jul 21 17:49:09.995095 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.994962 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bbb80d-84b0-4bfe-be6a-663922603694" containerName="main" Jul 21 17:49:09.995095 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.994969 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0aa8536-d2c6-41dd-9cf8-3a4081749dc5" containerName="test-curl-7639-140548771608592" Jul 21 17:49:09.998166 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:09.998095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.000614 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.000598 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b76jr\"/\"kube-root-ca.crt\"" Jul 21 17:49:10.001317 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.001298 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b76jr\"/\"openshift-service-ca.crt\"" Jul 21 17:49:10.001438 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.001330 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b76jr\"/\"default-dockercfg-4vv7m\"" Jul 21 17:49:10.009964 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.009942 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb"] Jul 21 17:49:10.041012 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.040984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-sys\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.041182 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.041034 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ph96\" (UniqueName: \"kubernetes.io/projected/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-kube-api-access-5ph96\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.041182 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.041054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-podres\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.041182 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.041168 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-proc\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.041305 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.041217 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-lib-modules\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142115 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-sys\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ph96\" (UniqueName: \"kubernetes.io/projected/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-kube-api-access-5ph96\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-podres\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142202 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-sys\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-proc\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142256 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-proc\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142302 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142261 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-lib-modules\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142525 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142333 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-lib-modules\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.142525 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.142344 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-podres\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.151011 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.150986 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ph96\" (UniqueName: \"kubernetes.io/projected/28e27d41-aa8b-4b40-bc8a-ab92e5cb8311-kube-api-access-5ph96\") pod \"perf-node-gather-daemonset-2q4sb\" (UID: \"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311\") " pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.308409 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.308317 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:10.440262 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:10.440236 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb"] Jul 21 17:49:10.442139 ip-10-0-134-97 kubenswrapper[2562]: W0721 17:49:10.442096 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod28e27d41_aa8b_4b40_bc8a_ab92e5cb8311.slice/crio-d9d794e19536e96a1e91e741ff239de694463a09a5e64eac14c107af19507125 WatchSource:0}: Error finding container d9d794e19536e96a1e91e741ff239de694463a09a5e64eac14c107af19507125: Status 404 returned error can't find the container with id d9d794e19536e96a1e91e741ff239de694463a09a5e64eac14c107af19507125 Jul 21 17:49:11.112556 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:11.112514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" event={"ID":"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311","Type":"ContainerStarted","Data":"dc2d7b83605b76edd71b0fcf7e910c463c2d51048c50925b123f5bfd39785a49"} Jul 21 17:49:11.112954 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:11.112570 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" event={"ID":"28e27d41-aa8b-4b40-bc8a-ab92e5cb8311","Type":"ContainerStarted","Data":"d9d794e19536e96a1e91e741ff239de694463a09a5e64eac14c107af19507125"} Jul 21 17:49:11.112954 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:11.112601 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:11.128856 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:11.128807 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" podStartSLOduration=2.12879065 podStartE2EDuration="2.12879065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:49:11.127369126 +0000 UTC m=+2819.241516112" watchObservedRunningTime="2026-07-21 17:49:11.12879065 +0000 UTC m=+2819.242937635" Jul 21 17:49:12.969894 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:12.969869 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ht659_e26fc102-988a-459c-a457-ab5918ef2002/dns/0.log" Jul 21 17:49:12.994391 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:12.994369 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ht659_e26fc102-988a-459c-a457-ab5918ef2002/kube-rbac-proxy/0.log" Jul 21 17:49:13.104333 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:13.104300 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-clg67_41c19cc9-d39e-424f-9c99-28f3d81a8522/dns-node-resolver/0.log" Jul 21 17:49:13.734248 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:13.734217 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5vkhj_5cfa7c62-ef41-4665-9794-751565d61390/node-ca/0.log" Jul 21 17:49:14.999578 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:14.999549 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66868cbd5-s6nbp_d486d602-ccaf-4538-bcf1-8b0a93a35fde/kube-auth-proxy/0.log" Jul 21 17:49:15.963491 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:15.963462 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nxvvr_4fd88f93-842d-46a9-92cf-15eda9be95d2/serve-healthcheck-canary/0.log" Jul 21 17:49:16.636335 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:16.636304 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rqs45_ba01ebad-5daf-479a-9a36-728feef6a3c9/kube-rbac-proxy/0.log" Jul 21 17:49:16.659489 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:16.659463 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rqs45_ba01ebad-5daf-479a-9a36-728feef6a3c9/exporter/0.log" Jul 21 17:49:16.683720 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:16.683692 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rqs45_ba01ebad-5daf-479a-9a36-728feef6a3c9/extractor/0.log" Jul 21 17:49:17.127409 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:17.127381 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b76jr/perf-node-gather-daemonset-2q4sb" Jul 21 17:49:19.281092 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:19.281060 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-8vldc_79fcc563-9de5-404b-8bc4-f8763ebdf35f/manager/0.log" Jul 21 17:49:19.422724 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:19.422695 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-npvmb_7102ce76-14d8-4e50-af46-33c80a7c8ac9/server/0.log" Jul 21 17:49:19.715919 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:19.715883 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk_fa65b591-bb96-4c3b-be95-cfd71f969bee/manager/0.log" Jul 21 17:49:19.739443 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:19.739415 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6cc55bc8b7-c7kwk_fa65b591-bb96-4c3b-be95-cfd71f969bee/copy-manifests/0.log" Jul 21 17:49:20.522300 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:20.522269 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66868cbd5-s6nbp_d486d602-ccaf-4538-bcf1-8b0a93a35fde/kube-auth-proxy/0.log" Jul 21 17:49:21.138686 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:21.138658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f64fdf99-2ckn7_d40707d6-2ef9-4048-9449-4503a16d2c4b/manager/0.log" Jul 21 17:49:22.089116 ip-10-0-134-97 kubenswrapper[2562]: I0721 17:49:22.089084 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66868cbd5-s6nbp_d486d602-ccaf-4538-bcf1-8b0a93a35fde/kube-auth-proxy/0.log"