Jul 06 22:46:47.740416 ip-10-0-134-52 systemd[1]: Starting Kubernetes Kubelet... Jul 06 22:46:48.228918 ip-10-0-134-52 systemd[1]: Started Kubernetes Kubelet. Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.151868 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156715 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156731 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156735 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156739 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156742 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156745 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156747 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:46:48.237196 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156750 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156753 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156755 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156758 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156761 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156764 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156768 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156773 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156776 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156789 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156792 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156795 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156798 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156800 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156803 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156806 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156809 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156812 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156814 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:46:48.238907 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156817 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156819 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156822 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156824 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156829 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156831 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156834 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156836 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156839 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156842 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156845 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156847 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156850 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156853 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156857 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156859 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156862 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156865 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156868 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156871 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:46:48.432038 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156873 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156876 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156878 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156881 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156890 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156893 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156896 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156898 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156901 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156905 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156910 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156913 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156916 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156919 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156922 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156924 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156927 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156929 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156932 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:46:48.432968 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156934 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156937 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156939 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156942 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156944 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156947 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156949 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156952 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156956 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156958 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156961 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156964 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156967 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156969 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156972 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156974 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156977 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156980 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156982 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156985 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:46:48.433645 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156988 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156991 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156994 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156996 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.156999 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157001 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157005 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157009 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157012 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157015 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157018 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157020 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157023 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157025 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157782 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157790 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157793 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157798 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157802 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157805 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:46:48.434243 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157808 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157811 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157814 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157817 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157820 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157823 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157825 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157828 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157831 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157833 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157836 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157839 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157842 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157845 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157848 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157850 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157853 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157855 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157858 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157861 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:46:48.434870 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157863 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157866 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157869 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157872 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157874 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157877 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157880 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157882 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157885 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157888 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157890 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157893 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157896 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157899 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157902 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157904 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157907 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157909 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157912 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157915 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:46:48.435533 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157918 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157920 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157923 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157925 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157930 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157934 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157936 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157939 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157944 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157947 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157950 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157952 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157955 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157958 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157960 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157964 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157966 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157970 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157977 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:46:48.436167 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157980 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157985 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157989 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157993 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.157997 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158001 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158004 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158007 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158010 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158013 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158015 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158018 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158020 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158023 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158025 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158028 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158030 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158033 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158036 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:46:48.436824 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158038 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158041 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158044 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158046 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158049 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158051 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158054 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158056 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158059 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158065 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158067 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158070 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158073 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158076 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.158079 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159184 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159204 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159209 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159213 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:46:48.437496 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159220 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159223 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159226 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159230 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159233 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159235 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159238 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159241 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159244 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159246 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159249 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159252 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159258 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159260 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159263 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159268 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159270 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159273 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159276 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159279 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:46:48.438106 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159283 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159287 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159292 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159298 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159302 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159304 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159307 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159310 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159313 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159317 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159320 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159322 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159325 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159328 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159331 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159334 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159339 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159341 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159344 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:46:48.438874 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159347 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159349 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159352 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159354 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159357 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159359 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159362 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159365 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159369 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159371 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159376 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159379 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159382 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159384 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159388 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159391 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159394 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159397 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159399 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159402 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:46:48.439456 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159405 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159410 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159413 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159416 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159419 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159421 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159424 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159427 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159429 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159432 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159435 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159437 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159440 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159442 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159447 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159450 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159453 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159455 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159458 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159460 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:46:48.440004 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159464 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159467 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159469 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159472 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159475 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159477 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159480 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159485 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159488 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159490 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159493 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159495 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159498 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159501 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159503 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.159506 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159590 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159611 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159642 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159648 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159652 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 22:46:48.440548 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159656 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159660 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159665 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159672 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159675 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159679 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159682 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159686 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159689 2579 flags.go:64] FLAG: --cgroup-root="" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159692 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159695 2579 flags.go:64] FLAG: --client-ca-file="" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159700 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159705 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159709 2579 flags.go:64] FLAG: --cluster-domain="" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159712 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159717 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159722 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159725 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159729 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159733 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159739 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159742 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159745 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159748 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 22:46:48.441153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159751 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159755 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159758 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159762 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159766 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159772 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159775 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159778 2579 flags.go:64] FLAG: --enable-server="true" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159781 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159786 2579 flags.go:64] FLAG: --event-burst="100" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159789 2579 flags.go:64] FLAG: --event-qps="50" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159793 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159796 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159801 2579 flags.go:64] FLAG: --eviction-hard="" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159806 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159809 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159812 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159815 2579 flags.go:64] FLAG: --eviction-soft="" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159819 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159823 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159827 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159830 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159835 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159840 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159843 2579 flags.go:64] FLAG: --feature-gates="" Jul 06 22:46:48.441803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159848 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159851 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159854 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159858 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159861 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159867 2579 flags.go:64] FLAG: --help="false" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159870 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159873 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159876 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159879 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159883 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159887 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159890 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159894 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159899 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159902 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159905 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159909 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159912 2579 flags.go:64] FLAG: --kube-reserved="" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159915 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159918 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159921 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159927 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159930 2579 flags.go:64] FLAG: --lock-file="" Jul 06 22:46:48.442536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159933 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159935 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159940 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159946 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159949 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159952 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159957 2579 flags.go:64] FLAG: --logging-format="text" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159963 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159967 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159970 2579 flags.go:64] FLAG: --manifest-url="" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159973 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159977 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159981 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159985 2579 flags.go:64] FLAG: --max-pods="110" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159988 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159994 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.159997 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160000 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160003 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160006 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160010 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160019 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160025 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160028 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 22:46:48.443196 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160031 2579 flags.go:64] FLAG: --pod-cidr="" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160034 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160041 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160044 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160047 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160050 2579 flags.go:64] FLAG: --port="10250" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160054 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160059 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09d20842d6333a919" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160063 2579 flags.go:64] FLAG: --qos-reserved="" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160066 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160069 2579 flags.go:64] FLAG: --register-node="true" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160073 2579 flags.go:64] FLAG: --register-with-taints="" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160077 2579 flags.go:64] FLAG: --registry-burst="10" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160080 2579 flags.go:64] FLAG: --registry-qps="5" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160085 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160090 2579 flags.go:64] FLAG: --reserved-memory="" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160094 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160098 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160101 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160104 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160108 2579 flags.go:64] FLAG: --runonce="false" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160111 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160114 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160120 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160136 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 22:46:48.443860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160139 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160142 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160145 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160149 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160152 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160156 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160159 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160164 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160168 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160171 2579 flags.go:64] FLAG: --system-cgroups="" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160174 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160181 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160184 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160187 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160193 2579 flags.go:64] FLAG: --tls-min-version="" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160196 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160199 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160202 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160207 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160211 2579 flags.go:64] FLAG: --v="2" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160215 2579 flags.go:64] FLAG: --version="false" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160221 2579 flags.go:64] FLAG: --vmodule="" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160227 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.160231 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 22:46:48.444862 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160440 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160446 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160449 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160452 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160456 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160459 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160461 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160464 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160467 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160470 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160473 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160475 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160481 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160484 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160487 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160490 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160493 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160496 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160498 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160501 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:46:48.445585 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160504 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160506 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160509 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160512 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160518 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160520 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160525 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160527 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160530 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160535 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160538 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160540 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160543 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160546 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160548 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160551 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160554 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160559 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160562 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160564 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:46:48.701950 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160567 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160570 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160572 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160575 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160578 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160580 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160583 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160586 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160589 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160594 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160597 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160599 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160602 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160605 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160607 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160610 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160612 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160615 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160619 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160622 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:46:48.703234 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160624 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160628 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160633 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160636 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160639 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160642 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160645 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160649 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160653 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160656 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160658 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160661 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160664 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160668 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160674 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160677 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160680 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160683 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160685 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:46:48.703904 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160688 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160691 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160694 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160697 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160699 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160702 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160705 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160707 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160712 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160715 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160718 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160722 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160724 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160727 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160731 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160734 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160737 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160739 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160742 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:46:48.704512 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.160745 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.161370 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.167924 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.167943 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.167963 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.167969 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168025 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168032 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168048 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168052 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168056 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168060 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168063 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168066 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:46:48.705064 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168069 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168071 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168074 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168077 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168081 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168084 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168087 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168090 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168092 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168095 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168097 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168101 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168103 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168106 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168108 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168111 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168114 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168116 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168119 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168135 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:46:48.705594 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168138 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168141 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168143 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168146 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168150 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168153 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168156 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168159 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168162 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168164 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168166 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168170 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168172 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168175 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168178 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168181 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168185 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168188 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168191 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:46:48.706159 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168195 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168198 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168201 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168204 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168207 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168210 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168213 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168215 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168218 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168221 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168223 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168226 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168229 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168232 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168235 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168238 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168241 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168244 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168247 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168249 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:46:48.706812 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168252 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168255 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168257 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168260 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168264 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168267 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168270 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168273 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168276 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168280 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168283 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168286 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168288 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168291 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168293 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168296 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168298 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168301 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168304 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168306 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:46:48.707399 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168309 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168312 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168314 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168317 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168323 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168326 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168329 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168331 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168334 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168337 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168339 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168342 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.168347 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168463 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168468 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:46:48.708025 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168471 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168473 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168477 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168480 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168483 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168486 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168490 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168493 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168496 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168498 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168501 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168504 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168506 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168509 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168512 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168514 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168517 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168520 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168522 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:46:48.708527 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168525 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168527 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168530 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168543 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168546 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168548 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168551 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168553 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168556 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168558 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168562 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168566 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168569 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168573 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168576 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168581 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168584 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168586 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168589 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:46:48.709115 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168593 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168596 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168598 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168601 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168604 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168606 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168609 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168612 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168615 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168617 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168620 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168622 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168625 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168628 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168630 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168633 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168636 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168639 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168642 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168645 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:46:48.709680 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168647 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168650 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168652 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168655 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168657 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168660 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168662 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168665 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168668 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168671 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168674 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168676 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168680 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168683 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168685 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168688 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168690 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168693 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168695 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168698 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:46:48.710298 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168700 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168703 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168705 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168708 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168710 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168713 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168716 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168718 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168721 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168724 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168727 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168730 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168732 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168734 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168737 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168740 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168743 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168746 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:46:48.710873 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:48.168749 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.168753 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.168899 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.177243 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.178457 2579 server.go:1019] "Starting client certificate rotation" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.178558 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.178590 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.180838 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.182229 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.195900 2579 log.go:25] "Validated CRI v1 runtime API" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.198062 2579 log.go:25] "Validated CRI v1 image API" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.198901 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 22:46:48.711456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.201883 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 df9c29a1-26a5-41cb-b0a8-9264ded9bd02:/dev/nvme0n1p3 e85ca26c-fabb-44e0-8e8d-9c2f45c2a68c:/dev/nvme0n1p4] Jul 06 22:46:48.711836 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.201905 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 22:46:48.711946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.207432 2579 manager.go:216] Machine: {Timestamp:2026-07-06 22:46:48.205953848 +0000 UTC m=+0.341678416 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089503 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec295b48c605e98afc45d0fa8464abd5 SystemUUID:ec295b48-c605-e98a-fc45-d0fa8464abd5 BootID:9b14ff0c-83ed-4336-93da-360404a29abf Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ad:67:0a:a1:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ad:67:0a:a1:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:48:cf:b9:f4:66 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 22:46:48.711946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.207536 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 22:46:48.711946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.207621 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 22:46:48.711946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.208525 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 22:46:48.711946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.208549 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-52.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 22:46:48.711946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.208939 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:46:48.711946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.209244 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.209255 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.209263 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.209863 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.211069 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214062 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214215 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214243 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214253 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214271 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214281 2579 kubelet.go:398] "Adding apiserver pod source" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214293 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214638 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.214652 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.220955 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.222844 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.222872 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224704 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224718 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224724 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224730 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224739 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224746 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224754 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224761 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224767 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224773 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224797 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224804 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224827 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.224832 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.228003 2579 server.go:1268] "Started kubelet" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.228058 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.228176 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.228249 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.229811 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.230779 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.230877 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-52.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.230895 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.231534 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 06 22:46:48.712288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.237238 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-52bq8" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.237272 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.236802 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-52.ec2.internal.18bfd4d988ac0e44 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-52.ec2.internal,UID:ip-10-0-134-52.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-52.ec2.internal,},FirstTimestamp:2026-07-06 22:46:48.227974724 +0000 UTC m=+0.363699270,LastTimestamp:2026-07-06 22:46:48.227974724 +0000 UTC m=+0.363699270,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-52.ec2.internal,}" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.242113 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.242138 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.242247 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.242669 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.242702 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.242767 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.242777 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.242768 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.244616 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-52.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.244650 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.244673 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.245091 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-52bq8" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.245464 2579 factory.go:55] Registering systemd factory Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.245484 2579 factory.go:223] Registration of the systemd container factory successfully Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.245768 2579 factory.go:153] Registering CRI-O factory Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.245782 2579 factory.go:223] Registration of the crio container factory successfully Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.245806 2579 factory.go:103] Registering Raw factory Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.245819 2579 manager.go:1196] Started watching for new ooms in manager Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.246455 2579 manager.go:349] Starting recovery of all containers Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.255742 2579 manager.go:354] Recovery completed Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.261034 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.263078 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.263114 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.263141 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.263656 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.263664 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.263682 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.263698 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.264375 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:46:48.713436 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.264397 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.264406 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.264883 2579 policy_none.go:49] "None policy: Start" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.264898 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.264908 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.266079 2579 policy_none.go:47] "Start" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.302735 2579 manager.go:337] "Starting Device Plugin manager" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.302769 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.302782 2579 server.go:87] "Starting device plugin registration server" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.303027 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.303038 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.303159 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.303246 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.303256 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.303810 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.303863 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.403554 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.404556 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.404586 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.404597 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.404622 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.413467 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.413489 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-52.ec2.internal\": node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.420004 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.421385 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.421418 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.421438 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.421467 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.423650 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.430971 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.524800 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"] Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.524912 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.526004 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.526038 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.526054 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.528695 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.528779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.528814 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:46:48.911551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.529457 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.529489 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.529500 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.529459 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.529561 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.529578 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.530804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.530834 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.531027 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.531626 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.531648 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.531660 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.544668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dc46d6abc3ef28f9f3665a9927f901c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"dc46d6abc3ef28f9f3665a9927f901c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.544700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc46d6abc3ef28f9f3665a9927f901c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"dc46d6abc3ef28f9f3665a9927f901c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.544723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8420a51f9e906ef04e32099f52a2c79-config\") pod \"kube-apiserver-proxy-ip-10-0-134-52.ec2.internal\" (UID: \"a8420a51f9e906ef04e32099f52a2c79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.566131 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-52.ec2.internal\" not found" node="ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.570837 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-52.ec2.internal\" not found" node="ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.632184 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.645838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dc46d6abc3ef28f9f3665a9927f901c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"dc46d6abc3ef28f9f3665a9927f901c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.645870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc46d6abc3ef28f9f3665a9927f901c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"dc46d6abc3ef28f9f3665a9927f901c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.645891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8420a51f9e906ef04e32099f52a2c79-config\") pod \"kube-apiserver-proxy-ip-10-0-134-52.ec2.internal\" (UID: \"a8420a51f9e906ef04e32099f52a2c79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.913575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.645950 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a8420a51f9e906ef04e32099f52a2c79-config\") pod \"kube-apiserver-proxy-ip-10-0-134-52.ec2.internal\" (UID: \"a8420a51f9e906ef04e32099f52a2c79\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.919338 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.645965 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dc46d6abc3ef28f9f3665a9927f901c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"dc46d6abc3ef28f9f3665a9927f901c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.919338 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.645947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dc46d6abc3ef28f9f3665a9927f901c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal\" (UID: \"dc46d6abc3ef28f9f3665a9927f901c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.919338 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.732384 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.919338 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.839032 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:48.919338 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.869758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:48.919338 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:48.874876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" Jul 06 22:46:49.150060 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:48.939328 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:49.150060 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:49.042877 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:49.150060 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.052978 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:46:49.150060 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.114217 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:46:49.150060 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:49.143524 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.178335 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.178488 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.178526 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.178525 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:49.244082 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-52.ec2.internal\" not found" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.247754 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.249660 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 22:41:48 +0000 UTC" deadline="2028-02-17 21:21:29.737464402 +0000 UTC" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.249687 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14182h34m40.487781791s" Jul 06 22:46:49.349747 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.343353 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" Jul 06 22:46:49.502623 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.355523 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:46:49.502623 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.356526 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" Jul 06 22:46:49.502623 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.368171 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:46:49.708842 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:49.534474 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8420a51f9e906ef04e32099f52a2c79.slice/crio-684cbc6135d48b75c33019549ae72ff93a794875efc223d87c8cd02f4e57445b WatchSource:0}: Error finding container 684cbc6135d48b75c33019549ae72ff93a794875efc223d87c8cd02f4e57445b: Status 404 returned error can't find the container with id 684cbc6135d48b75c33019549ae72ff93a794875efc223d87c8cd02f4e57445b Jul 06 22:46:49.708842 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:49.534864 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc46d6abc3ef28f9f3665a9927f901c6.slice/crio-89b73b51c172765d4f1fc9dfeb3a975a0472c7ab815cabe87eaa89d938b64cc6 WatchSource:0}: Error finding container 89b73b51c172765d4f1fc9dfeb3a975a0472c7ab815cabe87eaa89d938b64cc6: Status 404 returned error can't find the container with id 89b73b51c172765d4f1fc9dfeb3a975a0472c7ab815cabe87eaa89d938b64cc6 Jul 06 22:46:49.708842 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:49.540538 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.216736 2579 apiserver.go:52] "Watching apiserver" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.224218 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.224516 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4wdrm","openshift-image-registry/node-ca-nv2vk","openshift-network-diagnostics/network-check-target-jmt2k","openshift-network-operator/iptables-alerter-bh52b","kube-system/konnectivity-agent-6cj5p","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8","openshift-cluster-node-tuning-operator/tuned-bxlzt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal","openshift-multus/multus-9cppd","openshift-multus/multus-additional-cni-plugins-phs26","openshift-multus/network-metrics-daemon-cmkgb","openshift-ovn-kubernetes/ovnkube-node-rw68r","kube-system/global-pull-secret-syncer-v7ccm","kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal"] Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.228645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.231011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.233257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.233312 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.235472 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.237806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.240091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.242939 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.243062 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.245230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.245288 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.247592 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b507c59f-0773-4b0f-9d84-7e9aedb5561c-tmp-dir\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f6aec2-1b74-43cd-9a5d-840a409c64e0-konnectivity-ca\") pod \"konnectivity-agent-6cj5p\" (UID: \"74f6aec2-1b74-43cd-9a5d-840a409c64e0\") " pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzl8w\" (UniqueName: \"kubernetes.io/projected/aa1688f2-ab1e-4256-ab16-c93df1e71419-kube-api-access-nzl8w\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.294582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysctl-conf\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvh64\" (UniqueName: \"kubernetes.io/projected/296d7868-09eb-4708-bad8-1b568068cb62-kube-api-access-tvh64\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-cni-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-multus-certs\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-socket-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-sys\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-lib-modules\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-var-lib-kubelet\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-system-cni-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-kubelet\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkwzn\" (UniqueName: \"kubernetes.io/projected/b507c59f-0773-4b0f-9d84-7e9aedb5561c-kube-api-access-gkwzn\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-device-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/296d7868-09eb-4708-bad8-1b568068cb62-etc-tuned\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.295578 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-netns\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-etc-kubernetes\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-host\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-sys-fs\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254674 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysconfig\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254691 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-kubernetes\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-os-release\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4f7345a-0a74-4474-9ef1-d611034d3682-cni-binary-copy\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-cni-multus\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-run\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296d7868-09eb-4708-bad8-1b568068cb62-tmp\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-socket-dir-parent\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-hostroot\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.299837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-conf-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f6aec2-1b74-43cd-9a5d-840a409c64e0-agent-certs\") pod \"konnectivity-agent-6cj5p\" (UID: \"74f6aec2-1b74-43cd-9a5d-840a409c64e0\") " pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.254976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-registration-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-host\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-cnibin\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-cni-bin\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255117 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvbh\" (UniqueName: \"kubernetes.io/projected/efadaccb-16ab-4bf9-aa7d-93db206c15ab-kube-api-access-dcvbh\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-serviceca\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255172 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-systemd\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-k8s-cni-cncf-io\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-daemon-config\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255263 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvsc\" (UniqueName: \"kubernetes.io/projected/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-kube-api-access-bcvsc\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.300592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-modprobe-d\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysctl-d\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srlw8\" (UniqueName: \"kubernetes.io/projected/c4f7345a-0a74-4474-9ef1-d611034d3682-kube-api-access-srlw8\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.255370 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b507c59f-0773-4b0f-9d84-7e9aedb5561c-hosts-file\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.272115 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.272308 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.272369 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.272379 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.272378 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.273850 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pc5sr\"" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.273854 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 22:46:50.301369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.273849 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f7t4d\"" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-cnibin\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-serviceca\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1db32628-6834-406d-afb6-eb4f71d1568c-iptables-alerter-script\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysctl-d\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b507c59f-0773-4b0f-9d84-7e9aedb5561c-tmp-dir\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356435 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-cnibin\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f6aec2-1b74-43cd-9a5d-840a409c64e0-konnectivity-ca\") pod \"konnectivity-agent-6cj5p\" (UID: \"74f6aec2-1b74-43cd-9a5d-840a409c64e0\") " pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzl8w\" (UniqueName: \"kubernetes.io/projected/aa1688f2-ab1e-4256-ab16-c93df1e71419-kube-api-access-nzl8w\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356540 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysctl-d\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-cni-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-lib-modules\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-kubelet\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.402437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356679 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-cni-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.356686 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkwzn\" (UniqueName: \"kubernetes.io/projected/b507c59f-0773-4b0f-9d84-7e9aedb5561c-kube-api-access-gkwzn\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-etc-kubernetes\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356742 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-kubelet\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-etc-kubernetes\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.356793 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs podName:efadaccb-16ab-4bf9-aa7d-93db206c15ab nodeName:}" failed. No retries permitted until 2026-07-06 22:46:50.856743783 +0000 UTC m=+2.992468338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs") pod "network-metrics-daemon-cmkgb" (UID: "efadaccb-16ab-4bf9-aa7d-93db206c15ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-sys-fs\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356835 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-serviceca\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4f7345a-0a74-4474-9ef1-d611034d3682-cni-binary-copy\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-lib-modules\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356836 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-cni-multus\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.411106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356896 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-sys-fs\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1db32628-6834-406d-afb6-eb4f71d1568c-host-slash\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-cni-multus\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-cnibin\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.356986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-run\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f6aec2-1b74-43cd-9a5d-840a409c64e0-agent-certs\") pod \"konnectivity-agent-6cj5p\" (UID: \"74f6aec2-1b74-43cd-9a5d-840a409c64e0\") " pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-registration-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357044 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-run\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-cni-bin\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-registration-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-var-lib-cni-bin\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357162 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvbh\" (UniqueName: \"kubernetes.io/projected/efadaccb-16ab-4bf9-aa7d-93db206c15ab-kube-api-access-dcvbh\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-systemd\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357194 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b507c59f-0773-4b0f-9d84-7e9aedb5561c-tmp-dir\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-k8s-cni-cncf-io\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.412631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-systemd\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-k8s-cni-cncf-io\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-daemon-config\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357345 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvsc\" (UniqueName: \"kubernetes.io/projected/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-kube-api-access-bcvsc\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-modprobe-d\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srlw8\" (UniqueName: \"kubernetes.io/projected/c4f7345a-0a74-4474-9ef1-d611034d3682-kube-api-access-srlw8\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b507c59f-0773-4b0f-9d84-7e9aedb5561c-hosts-file\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysctl-conf\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-modprobe-d\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvh64\" (UniqueName: \"kubernetes.io/projected/296d7868-09eb-4708-bad8-1b568068cb62-kube-api-access-tvh64\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357508 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b507c59f-0773-4b0f-9d84-7e9aedb5561c-hosts-file\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-multus-certs\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.413418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-binary-copy\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-socket-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357575 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-multus-certs\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysctl-conf\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-sys\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-var-lib-kubelet\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-system-cni-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-socket-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357711 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-sys\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j74ds\" (UniqueName: \"kubernetes.io/projected/1db32628-6834-406d-afb6-eb4f71d1568c-kube-api-access-j74ds\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-os-release\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357763 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-var-lib-kubelet\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357765 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-system-cni-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-device-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/296d7868-09eb-4708-bad8-1b568068cb62-etc-tuned\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.415368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357824 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-netns\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-system-cni-dir\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-host\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357872 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-device-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357890 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-host-run-netns\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysconfig\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357917 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-host\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-kubernetes\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-sysconfig\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-etc-kubernetes\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.357996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-os-release\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358034 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-os-release\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296d7868-09eb-4708-bad8-1b568068cb62-tmp\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-socket-dir-parent\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-hostroot\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.416115 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-conf-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-hostroot\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95795\" (UniqueName: \"kubernetes.io/projected/a858fa8c-dc0c-4990-b370-513310efb011-kube-api-access-95795\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-conf-dir\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-host\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358161 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-socket-dir-parent\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358138 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/296d7868-09eb-4708-bad8-1b568068cb62-host\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.358300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aa1688f2-ab1e-4256-ab16-c93df1e71419-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.361153 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/296d7868-09eb-4708-bad8-1b568068cb62-etc-tuned\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.361191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/296d7868-09eb-4708-bad8-1b568068cb62-tmp\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.517722 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-binary-copy\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j74ds\" (UniqueName: \"kubernetes.io/projected/1db32628-6834-406d-afb6-eb4f71d1568c-kube-api-access-j74ds\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-os-release\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-system-cni-dir\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.458894 2579 configmap.go:193] Couldn't get configMap openshift-multus/default-cni-sysctl-allowlist: object "openshift-multus"/"default-cni-sysctl-allowlist" not registered Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-tuning-conf-dir\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.458961 2579 configmap.go:193] Couldn't get configMap openshift-multus/whereabouts-flatfile-config: object "openshift-multus"/"whereabouts-flatfile-config" not registered Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.458977 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-sysctl-allowlist podName:a858fa8c-dc0c-4990-b370-513310efb011 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:50.958958102 +0000 UTC m=+3.094682638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cni-sysctl-allowlist" (UniqueName: "kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-sysctl-allowlist") pod "multus-additional-cni-plugins-phs26" (UID: "a858fa8c-dc0c-4990-b370-513310efb011") : object "openshift-multus"/"default-cni-sysctl-allowlist" not registered Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-os-release\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.458897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.459020 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-whereabouts-flatfile-configmap podName:a858fa8c-dc0c-4990-b370-513310efb011 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:50.959011063 +0000 UTC m=+3.094735598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "whereabouts-flatfile-configmap" (UniqueName: "kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-whereabouts-flatfile-configmap") pod "multus-additional-cni-plugins-phs26" (UID: "a858fa8c-dc0c-4990-b370-513310efb011") : object "openshift-multus"/"whereabouts-flatfile-config" not registered Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.459024 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-system-cni-dir\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.459038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95795\" (UniqueName: \"kubernetes.io/projected/a858fa8c-dc0c-4990-b370-513310efb011-kube-api-access-95795\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.518565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.459071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1db32628-6834-406d-afb6-eb4f71d1568c-iptables-alerter-script\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.520032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.459117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1db32628-6834-406d-afb6-eb4f71d1568c-host-slash\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.520032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.459172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-cnibin\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.520032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.459198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1db32628-6834-406d-afb6-eb4f71d1568c-host-slash\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.520032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.459220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a858fa8c-dc0c-4990-b370-513310efb011-cnibin\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.541667 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.540960 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 22:46:50.541667 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.541257 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kjn6w\"" Jul 06 22:46:50.541667 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.541360 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 22:46:50.541667 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.541504 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-86knk\"" Jul 06 22:46:50.541667 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.541568 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 22:46:50.542099 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.541799 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.542806 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.542381 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 22:46:50.542806 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.542523 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:46:50.542806 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.542585 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 22:46:50.542994 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.542886 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 22:46:50.542994 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.542906 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 22:46:50.543418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.543098 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t96cf\"" Jul 06 22:46:50.543877 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.543857 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 22:46:50.544182 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544158 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kth7n\"" Jul 06 22:46:50.544297 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544231 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 22:46:50.545267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544439 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:46:50.545267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544474 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7www5\"" Jul 06 22:46:50.545267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544485 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 22:46:50.545267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 22:46:50.545267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544734 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.545267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.544821 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:46:50.546134 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.545984 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 22:46:50.546134 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.545990 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9fkc5\"" Jul 06 22:46:50.546134 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.546029 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 22:46:50.547136 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.547098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/74f6aec2-1b74-43cd-9a5d-840a409c64e0-konnectivity-ca\") pod \"konnectivity-agent-6cj5p\" (UID: \"74f6aec2-1b74-43cd-9a5d-840a409c64e0\") " pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.547819 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.547797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c4f7345a-0a74-4474-9ef1-d611034d3682-multus-daemon-config\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.556883 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.555828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.556883 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.555928 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:46:50.556883 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.555946 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2z7cj\"" Jul 06 22:46:50.556883 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.556092 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 22:46:50.556883 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.556110 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 22:46:50.556883 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.556341 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 22:46:50.556883 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.556429 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 22:46:50.557448 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.557416 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1db32628-6834-406d-afb6-eb4f71d1568c-iptables-alerter-script\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.557899 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.557877 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 22:46:50.557977 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.557939 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c4f7345a-0a74-4474-9ef1-d611034d3682-cni-binary-copy\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.558195 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.558159 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 22:46:50.558533 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.558499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-binary-copy\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.559586 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.559027 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:46:50.559586 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.559050 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:46:50.559586 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.559071 2579 projected.go:196] Error preparing data for projected volume kube-api-access-bspnl for pod openshift-network-diagnostics/network-check-target-jmt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:50.559586 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.559172 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl podName:2c329ad7-e344-4011-85b7-acb16c4fdb13 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:51.05915123 +0000 UTC m=+3.194875767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bspnl" (UniqueName: "kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl") pod "network-check-target-jmt2k" (UID: "2c329ad7-e344-4011-85b7-acb16c4fdb13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:50.559586 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-systemd\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.559586 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.559586 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-log-socket\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-node-log\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-slash\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-var-lib-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559752 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j74ds\" (UniqueName: \"kubernetes.io/projected/1db32628-6834-406d-afb6-eb4f71d1568c-kube-api-access-j74ds\") pod \"iptables-alerter-bh52b\" (UID: \"1db32628-6834-406d-afb6-eb4f71d1568c\") " pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkwzn\" (UniqueName: \"kubernetes.io/projected/b507c59f-0773-4b0f-9d84-7e9aedb5561c-kube-api-access-gkwzn\") pod \"node-resolver-4wdrm\" (UID: \"b507c59f-0773-4b0f-9d84-7e9aedb5561c\") " pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-ovn\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5q5p\" (UniqueName: \"kubernetes.io/projected/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-kube-api-access-l5q5p\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovnkube-config\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.559986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-env-overrides\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560044 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovnkube-script-lib\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-kubelet\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-cni-netd\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-run-netns\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-etc-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovn-node-metrics-cert\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-systemd-units\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.560679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.560360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-cni-bin\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.562409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.562135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" event={"ID":"dc46d6abc3ef28f9f3665a9927f901c6","Type":"ContainerStarted","Data":"89b73b51c172765d4f1fc9dfeb3a975a0472c7ab815cabe87eaa89d938b64cc6"} Jul 06 22:46:50.562409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.562249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" event={"ID":"a8420a51f9e906ef04e32099f52a2c79","Type":"ContainerStarted","Data":"684cbc6135d48b75c33019549ae72ff93a794875efc223d87c8cd02f4e57445b"} Jul 06 22:46:50.565226 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.564984 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/74f6aec2-1b74-43cd-9a5d-840a409c64e0-agent-certs\") pod \"konnectivity-agent-6cj5p\" (UID: \"74f6aec2-1b74-43cd-9a5d-840a409c64e0\") " pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.565226 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.565021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95795\" (UniqueName: \"kubernetes.io/projected/a858fa8c-dc0c-4990-b370-513310efb011-kube-api-access-95795\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.565465 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.565441 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srlw8\" (UniqueName: \"kubernetes.io/projected/c4f7345a-0a74-4474-9ef1-d611034d3682-kube-api-access-srlw8\") pod \"multus-9cppd\" (UID: \"c4f7345a-0a74-4474-9ef1-d611034d3682\") " pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.565550 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.565531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvbh\" (UniqueName: \"kubernetes.io/projected/efadaccb-16ab-4bf9-aa7d-93db206c15ab-kube-api-access-dcvbh\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:50.565626 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.565552 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvh64\" (UniqueName: \"kubernetes.io/projected/296d7868-09eb-4708-bad8-1b568068cb62-kube-api-access-tvh64\") pod \"tuned-bxlzt\" (UID: \"296d7868-09eb-4708-bad8-1b568068cb62\") " pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.565626 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.565534 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvsc\" (UniqueName: \"kubernetes.io/projected/f4f73e02-0d1f-4c9e-bb6d-052c1a075e94-kube-api-access-bcvsc\") pod \"node-ca-nv2vk\" (UID: \"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94\") " pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.565732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.565646 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:46:50.566810 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.566736 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzl8w\" (UniqueName: \"kubernetes.io/projected/aa1688f2-ab1e-4256-ab16-c93df1e71419-kube-api-access-nzl8w\") pod \"aws-ebs-csi-driver-node-v8zk8\" (UID: \"aa1688f2-ab1e-4256-ab16-c93df1e71419\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.569594 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.569570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" Jul 06 22:46:50.576903 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.576874 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w4ffd" Jul 06 22:46:50.582453 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.582423 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296d7868_09eb_4708_bad8_1b568068cb62.slice/crio-200428cbd503414706dda3e991ff9673c9ad010bb408738fd974a511f6e4741b WatchSource:0}: Error finding container 200428cbd503414706dda3e991ff9673c9ad010bb408738fd974a511f6e4741b: Status 404 returned error can't find the container with id 200428cbd503414706dda3e991ff9673c9ad010bb408738fd974a511f6e4741b Jul 06 22:46:50.586537 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.586514 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w4ffd" Jul 06 22:46:50.643409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.643380 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 22:46:50.660954 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.660855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovnkube-config\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.660954 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.660904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-kubelet-config\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.660954 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.660944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-env-overrides\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.660971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovnkube-script-lib\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661015 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-kubelet\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-cni-netd\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-run-netns\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-etc-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovn-node-metrics-cert\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-cni-netd\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661218 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-etc-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-kubelet\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-run-netns\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661267 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-systemd-units\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-cni-bin\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-systemd\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-systemd-units\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-dbus\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-log-socket\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-cni-bin\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661586 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-systemd\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661588 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-node-log\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-node-log\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.661853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-slash\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-var-lib-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-host-slash\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-log-socket\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-ovn\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661726 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-var-lib-openvswitch\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q5p\" (UniqueName: \"kubernetes.io/projected/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-kube-api-access-l5q5p\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-run-ovn\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovnkube-config\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovnkube-script-lib\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.662467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.661981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-env-overrides\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.664023 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.663978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-ovn-node-metrics-cert\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.673394 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.673373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q5p\" (UniqueName: \"kubernetes.io/projected/e689f81a-bbf3-4c0b-8b1b-86b6c34f394f-kube-api-access-l5q5p\") pod \"ovnkube-node-rw68r\" (UID: \"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.762881 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.762694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.762881 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.762747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-kubelet-config\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.762881 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.762833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-kubelet-config\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.762881 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.762858 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:50.763282 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.762977 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret podName:4a33cb94-f91f-4a25-85d5-850bd2ab71e3 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:51.262953669 +0000 UTC m=+3.398678215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret") pod "global-pull-secret-syncer-v7ccm" (UID: "4a33cb94-f91f-4a25-85d5-850bd2ab71e3") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:50.763355 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.763287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-dbus\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.763497 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.763476 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-dbus\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:50.839080 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.839033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9cppd" Jul 06 22:46:50.840067 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.840043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bh52b" Jul 06 22:46:50.844409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.844383 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nv2vk" Jul 06 22:46:50.851592 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.851563 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f7345a_0a74_4474_9ef1_d611034d3682.slice/crio-4f80fb62fdcc51be01de1876cbbc51c9ee9ffb7597ea81dca2066e6b6686a7a7 WatchSource:0}: Error finding container 4f80fb62fdcc51be01de1876cbbc51c9ee9ffb7597ea81dca2066e6b6686a7a7: Status 404 returned error can't find the container with id 4f80fb62fdcc51be01de1876cbbc51c9ee9ffb7597ea81dca2066e6b6686a7a7 Jul 06 22:46:50.853378 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.853355 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wdrm" Jul 06 22:46:50.857049 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.856638 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db32628_6834_406d_afb6_eb4f71d1568c.slice/crio-73ced6b45c3c9e71fbce71dfb13b4e09e5bf6cc673779a67e73eec44b41dad88 WatchSource:0}: Error finding container 73ced6b45c3c9e71fbce71dfb13b4e09e5bf6cc673779a67e73eec44b41dad88: Status 404 returned error can't find the container with id 73ced6b45c3c9e71fbce71dfb13b4e09e5bf6cc673779a67e73eec44b41dad88 Jul 06 22:46:50.857885 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.857800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:46:50.859503 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.859436 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f73e02_0d1f_4c9e_bb6d_052c1a075e94.slice/crio-8d0cc19414439f50d9d5567890d4b6ad9cf639cf7c47f153941e67f1e19782af WatchSource:0}: Error finding container 8d0cc19414439f50d9d5567890d4b6ad9cf639cf7c47f153941e67f1e19782af: Status 404 returned error can't find the container with id 8d0cc19414439f50d9d5567890d4b6ad9cf639cf7c47f153941e67f1e19782af Jul 06 22:46:50.868155 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.864842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:50.868155 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.864980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" Jul 06 22:46:50.868155 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.865077 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:50.868155 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:50.865170 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs podName:efadaccb-16ab-4bf9-aa7d-93db206c15ab nodeName:}" failed. No retries permitted until 2026-07-06 22:46:51.865132036 +0000 UTC m=+4.000856587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs") pod "network-metrics-daemon-cmkgb" (UID: "efadaccb-16ab-4bf9-aa7d-93db206c15ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:50.873170 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.873142 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb507c59f_0773_4b0f_9d84_7e9aedb5561c.slice/crio-9757d06cb06dedf0c6cf0e5dd077782b92ff7b7cbc2467c3cfdd9b330d307678 WatchSource:0}: Error finding container 9757d06cb06dedf0c6cf0e5dd077782b92ff7b7cbc2467c3cfdd9b330d307678: Status 404 returned error can't find the container with id 9757d06cb06dedf0c6cf0e5dd077782b92ff7b7cbc2467c3cfdd9b330d307678 Jul 06 22:46:50.875256 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.875224 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f6aec2_1b74_43cd_9a5d_840a409c64e0.slice/crio-a9e26de2dcc45dee3c2cf27e547b88d6809733f9c82bbc38c208d509117935a2 WatchSource:0}: Error finding container a9e26de2dcc45dee3c2cf27e547b88d6809733f9c82bbc38c208d509117935a2: Status 404 returned error can't find the container with id a9e26de2dcc45dee3c2cf27e547b88d6809733f9c82bbc38c208d509117935a2 Jul 06 22:46:50.881758 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.881733 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1688f2_ab1e_4256_ab16_c93df1e71419.slice/crio-b2c9c0e19f0f869f1674e52decf89755469ceba08fadfe3f8c26933ac7a88674 WatchSource:0}: Error finding container b2c9c0e19f0f869f1674e52decf89755469ceba08fadfe3f8c26933ac7a88674: Status 404 returned error can't find the container with id b2c9c0e19f0f869f1674e52decf89755469ceba08fadfe3f8c26933ac7a88674 Jul 06 22:46:50.899734 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.899702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:46:50.911848 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:46:50.911774 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode689f81a_bbf3_4c0b_8b1b_86b6c34f394f.slice/crio-c77c88a55e238d5e8fdc8b777477f568ce007ce5c1066ae6e97693a84447b13c WatchSource:0}: Error finding container c77c88a55e238d5e8fdc8b777477f568ce007ce5c1066ae6e97693a84447b13c: Status 404 returned error can't find the container with id c77c88a55e238d5e8fdc8b777477f568ce007ce5c1066ae6e97693a84447b13c Jul 06 22:46:50.966061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.966024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.966247 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.966138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.966792 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.966763 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:50.966932 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:50.966765 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a858fa8c-dc0c-4990-b370-513310efb011-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-phs26\" (UID: \"a858fa8c-dc0c-4990-b370-513310efb011\") " pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:51.066936 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.066895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:51.067158 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.067123 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:46:51.067158 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.067148 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:46:51.067289 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.067183 2579 projected.go:196] Error preparing data for projected volume kube-api-access-bspnl for pod openshift-network-diagnostics/network-check-target-jmt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:51.067289 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.067254 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl podName:2c329ad7-e344-4011-85b7-acb16c4fdb13 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:52.067233649 +0000 UTC m=+4.202958187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bspnl" (UniqueName: "kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl") pod "network-check-target-jmt2k" (UID: "2c329ad7-e344-4011-85b7-acb16c4fdb13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:51.192946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.192859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-phs26" Jul 06 22:46:51.268907 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.268864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:51.269108 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.269005 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:51.269108 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.269079 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret podName:4a33cb94-f91f-4a25-85d5-850bd2ab71e3 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:52.269057897 +0000 UTC m=+4.404782447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret") pod "global-pull-secret-syncer-v7ccm" (UID: "4a33cb94-f91f-4a25-85d5-850bd2ab71e3") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:51.321387 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.321349 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:46:51.431146 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.431101 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" event={"ID":"aa1688f2-ab1e-4256-ab16-c93df1e71419","Type":"ContainerStarted","Data":"b2c9c0e19f0f869f1674e52decf89755469ceba08fadfe3f8c26933ac7a88674"} Jul 06 22:46:51.432439 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.432403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wdrm" event={"ID":"b507c59f-0773-4b0f-9d84-7e9aedb5561c","Type":"ContainerStarted","Data":"9757d06cb06dedf0c6cf0e5dd077782b92ff7b7cbc2467c3cfdd9b330d307678"} Jul 06 22:46:51.433751 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.433728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"c77c88a55e238d5e8fdc8b777477f568ce007ce5c1066ae6e97693a84447b13c"} Jul 06 22:46:51.434998 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.434973 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6cj5p" event={"ID":"74f6aec2-1b74-43cd-9a5d-840a409c64e0","Type":"ContainerStarted","Data":"a9e26de2dcc45dee3c2cf27e547b88d6809733f9c82bbc38c208d509117935a2"} Jul 06 22:46:51.436206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.436181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nv2vk" event={"ID":"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94","Type":"ContainerStarted","Data":"8d0cc19414439f50d9d5567890d4b6ad9cf639cf7c47f153941e67f1e19782af"} Jul 06 22:46:51.437355 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.437321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bh52b" event={"ID":"1db32628-6834-406d-afb6-eb4f71d1568c","Type":"ContainerStarted","Data":"73ced6b45c3c9e71fbce71dfb13b4e09e5bf6cc673779a67e73eec44b41dad88"} Jul 06 22:46:51.438662 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.438641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cppd" event={"ID":"c4f7345a-0a74-4474-9ef1-d611034d3682","Type":"ContainerStarted","Data":"4f80fb62fdcc51be01de1876cbbc51c9ee9ffb7597ea81dca2066e6b6686a7a7"} Jul 06 22:46:51.440036 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.440016 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" event={"ID":"296d7868-09eb-4708-bad8-1b568068cb62","Type":"ContainerStarted","Data":"200428cbd503414706dda3e991ff9673c9ad010bb408738fd974a511f6e4741b"} Jul 06 22:46:51.590654 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.590505 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:41:50 +0000 UTC" deadline="2028-01-22 18:31:41.066421904 +0000 UTC" Jul 06 22:46:51.590654 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.590546 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13555h44m49.475881193s" Jul 06 22:46:51.636655 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.636429 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:46:51.876564 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:51.875872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:51.876564 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.876044 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:51.876564 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:51.876110 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs podName:efadaccb-16ab-4bf9-aa7d-93db206c15ab nodeName:}" failed. No retries permitted until 2026-07-06 22:46:53.876089496 +0000 UTC m=+6.011814046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs") pod "network-metrics-daemon-cmkgb" (UID: "efadaccb-16ab-4bf9-aa7d-93db206c15ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:52.078864 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:52.078031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:52.078864 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.078272 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:46:52.078864 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.078294 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:46:52.078864 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.078308 2579 projected.go:196] Error preparing data for projected volume kube-api-access-bspnl for pod openshift-network-diagnostics/network-check-target-jmt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:52.078864 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.078370 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl podName:2c329ad7-e344-4011-85b7-acb16c4fdb13 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:54.078350585 +0000 UTC m=+6.214075126 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bspnl" (UniqueName: "kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl") pod "network-check-target-jmt2k" (UID: "2c329ad7-e344-4011-85b7-acb16c4fdb13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:52.280563 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:52.279925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:52.280563 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.280070 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:52.280563 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.280137 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret podName:4a33cb94-f91f-4a25-85d5-850bd2ab71e3 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:54.280117241 +0000 UTC m=+6.415841777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret") pod "global-pull-secret-syncer-v7ccm" (UID: "4a33cb94-f91f-4a25-85d5-850bd2ab71e3") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:52.425263 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:52.424493 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:52.425263 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.424640 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:46:52.425263 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:52.425047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:52.425263 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.425158 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:46:52.426102 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:52.426049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:52.426216 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:52.426190 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:46:52.591690 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:52.591562 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:41:50 +0000 UTC" deadline="2028-02-12 02:02:21.180311953 +0000 UTC" Jul 06 22:46:52.591690 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:52.591607 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14043h15m28.588710189s" Jul 06 22:46:53.452296 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:53.451900 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerStarted","Data":"c248507dddb309f3263b37c73bbffa16a41a64ce9c98a2b4e84af2ec6c19b009"} Jul 06 22:46:53.897431 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:53.897341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:53.897893 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:53.897514 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:53.897893 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:53.897584 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs podName:efadaccb-16ab-4bf9-aa7d-93db206c15ab nodeName:}" failed. No retries permitted until 2026-07-06 22:46:57.897560803 +0000 UTC m=+10.033285342 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs") pod "network-metrics-daemon-cmkgb" (UID: "efadaccb-16ab-4bf9-aa7d-93db206c15ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:54.099848 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:54.099175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:54.099848 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.099386 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:46:54.099848 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.099405 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:46:54.099848 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.099419 2579 projected.go:196] Error preparing data for projected volume kube-api-access-bspnl for pod openshift-network-diagnostics/network-check-target-jmt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:54.099848 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.099479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl podName:2c329ad7-e344-4011-85b7-acb16c4fdb13 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:58.099458796 +0000 UTC m=+10.235183333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bspnl" (UniqueName: "kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl") pod "network-check-target-jmt2k" (UID: "2c329ad7-e344-4011-85b7-acb16c4fdb13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:54.301634 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:54.301528 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:54.301813 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.301679 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:54.301813 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.301746 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret podName:4a33cb94-f91f-4a25-85d5-850bd2ab71e3 nodeName:}" failed. No retries permitted until 2026-07-06 22:46:58.301725743 +0000 UTC m=+10.437450281 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret") pod "global-pull-secret-syncer-v7ccm" (UID: "4a33cb94-f91f-4a25-85d5-850bd2ab71e3") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:54.422318 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:54.422283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:54.422483 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.422422 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:46:54.423297 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:54.422852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:54.423297 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.422964 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:46:54.423297 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:54.423114 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:54.423297 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:54.423219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:46:56.422586 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:56.422550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:56.423080 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:56.422778 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:46:56.423177 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:56.423153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:56.423301 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:56.423266 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:46:56.423374 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:56.423311 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:56.423423 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:56.423380 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:46:57.931777 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:57.931740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:57.932198 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:57.931909 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:57.932198 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:57.931979 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs podName:efadaccb-16ab-4bf9-aa7d-93db206c15ab nodeName:}" failed. No retries permitted until 2026-07-06 22:47:05.931959336 +0000 UTC m=+18.067683881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs") pod "network-metrics-daemon-cmkgb" (UID: "efadaccb-16ab-4bf9-aa7d-93db206c15ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:46:58.134230 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:58.133505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:58.134230 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.133729 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:46:58.134230 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.133753 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:46:58.134230 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.133767 2579 projected.go:196] Error preparing data for projected volume kube-api-access-bspnl for pod openshift-network-diagnostics/network-check-target-jmt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:58.134230 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.133827 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl podName:2c329ad7-e344-4011-85b7-acb16c4fdb13 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:06.133808025 +0000 UTC m=+18.269532560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bspnl" (UniqueName: "kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl") pod "network-check-target-jmt2k" (UID: "2c329ad7-e344-4011-85b7-acb16c4fdb13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:46:58.335629 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:58.335582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:58.335801 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.335725 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:58.335801 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.335793 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret podName:4a33cb94-f91f-4a25-85d5-850bd2ab71e3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:06.335775496 +0000 UTC m=+18.471500043 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret") pod "global-pull-secret-syncer-v7ccm" (UID: "4a33cb94-f91f-4a25-85d5-850bd2ab71e3") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:46:58.423401 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:58.423172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:46:58.423401 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:58.423172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:46:58.423401 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.423278 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:46:58.423401 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.423366 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:46:58.423401 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:46:58.423179 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:46:58.423715 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:46:58.423522 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:00.422112 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:00.422076 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:00.422573 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:00.422221 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:00.422630 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:00.422584 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:00.422699 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:00.422686 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:00.422836 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:00.422815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:00.422961 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:00.422940 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:02.421885 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:02.421852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:02.422365 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:02.421853 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:02.422365 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:02.421995 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:02.422365 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:02.422077 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:02.422365 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:02.421857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:02.422365 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:02.422216 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:04.422533 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:04.422494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:04.422975 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:04.422501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:04.422975 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:04.422619 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:04.422975 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:04.422501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:04.422975 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:04.422704 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:04.422975 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:04.422803 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:05.992353 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:05.992322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:05.992757 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:05.992482 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:47:05.992757 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:05.992546 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs podName:efadaccb-16ab-4bf9-aa7d-93db206c15ab nodeName:}" failed. No retries permitted until 2026-07-06 22:47:21.992527465 +0000 UTC m=+34.128252016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs") pod "network-metrics-daemon-cmkgb" (UID: "efadaccb-16ab-4bf9-aa7d-93db206c15ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:47:06.194304 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:06.194264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:06.194464 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.194402 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:47:06.194464 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.194424 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:47:06.194464 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.194433 2579 projected.go:196] Error preparing data for projected volume kube-api-access-bspnl for pod openshift-network-diagnostics/network-check-target-jmt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:47:06.194583 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.194493 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl podName:2c329ad7-e344-4011-85b7-acb16c4fdb13 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:22.19447509 +0000 UTC m=+34.330199629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bspnl" (UniqueName: "kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl") pod "network-check-target-jmt2k" (UID: "2c329ad7-e344-4011-85b7-acb16c4fdb13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:47:06.395888 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:06.395798 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:06.396033 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.395966 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:47:06.396033 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.396023 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret podName:4a33cb94-f91f-4a25-85d5-850bd2ab71e3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:22.396008681 +0000 UTC m=+34.531733219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret") pod "global-pull-secret-syncer-v7ccm" (UID: "4a33cb94-f91f-4a25-85d5-850bd2ab71e3") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:47:06.422782 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:06.422748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:06.422782 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:06.422776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:06.422993 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:06.422877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:06.422993 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.422894 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:06.423112 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.422997 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:06.423112 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:06.423083 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:08.423258 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:08.423164 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:08.423736 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:08.423279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:08.423736 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:08.423295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:08.423736 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:08.423280 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:08.423736 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:08.423421 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:08.423736 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:08.423504 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:10.422233 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.422203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:10.422684 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.422210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:10.422684 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:10.422329 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:10.422684 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.422354 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:10.422684 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:10.422450 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:10.422684 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:10.422537 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:10.486372 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.486324 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9cppd" event={"ID":"c4f7345a-0a74-4474-9ef1-d611034d3682","Type":"ContainerStarted","Data":"dd1ab9593e53bb117ebdf1a6a2fecb2c6e70ced7c55a4ea8dc8139b8696b4385"} Jul 06 22:47:10.487849 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.487823 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" event={"ID":"296d7868-09eb-4708-bad8-1b568068cb62","Type":"ContainerStarted","Data":"9f1e9959a107d8e6f656ac9d292ee95ca7cc6eaac8075c3a7b12507e4c688c81"} Jul 06 22:47:10.489306 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.489284 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" event={"ID":"a8420a51f9e906ef04e32099f52a2c79","Type":"ContainerStarted","Data":"a0e2ac4aaa2812bd58bc2bb10319702a9cefcf642f9acae459c4c992844f7533"} Jul 06 22:47:10.545053 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.544998 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9cppd" podStartSLOduration=3.313109946 podStartE2EDuration="22.544979802s" totalImagesPullingTime="19.231869856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:10.520297828 +0000 UTC m=+22.656022380" watchObservedRunningTime="2026-07-06 22:47:10.544979802 +0000 UTC m=+22.680704358" Jul 06 22:47:10.545846 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:10.545599 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-52.ec2.internal" podStartSLOduration=21.54558749 podStartE2EDuration="21.54558749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:10.544497759 +0000 UTC m=+22.680222332" watchObservedRunningTime="2026-07-06 22:47:10.54558749 +0000 UTC m=+22.681312050" Jul 06 22:47:11.492989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.492728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wdrm" event={"ID":"b507c59f-0773-4b0f-9d84-7e9aedb5561c","Type":"ContainerStarted","Data":"af611666ce3076437ab591b2f3c8ffd9dbccb237aade4b0ca5370f4785ca4474"} Jul 06 22:47:11.495381 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495360 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:47:11.495683 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495661 2579 generic.go:356] "Generic (PLEG): container finished" podID="e689f81a-bbf3-4c0b-8b1b-86b6c34f394f" containerID="a36a953d220352539d45bc2686a20e742cc13eacb2ef5a5dc184ddb2d0ad0ace" exitCode=1 Jul 06 22:47:11.495750 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495731 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"1e37f6550c746addcdd6ed62926335d90ac67d69b2b848d96cb1d8dafbb075c5"} Jul 06 22:47:11.495807 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495790 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"1931aadbcd1bcb9bb815df44c5ada91d8da27223878bd1d776199bdd967c5de7"} Jul 06 22:47:11.495863 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495830 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"fc2c45d710ffe09a92046a82122a88a9e55b57f2f25e7e51a62ae2ec46ab8ca4"} Jul 06 22:47:11.495863 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495858 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"856f3235624a08b61f5ce74ceaf92abcc08fded6b24e772b9c5ec144d4255660"} Jul 06 22:47:11.495949 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495893 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerDied","Data":"a36a953d220352539d45bc2686a20e742cc13eacb2ef5a5dc184ddb2d0ad0ace"} Jul 06 22:47:11.495949 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.495924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"c79a3b0b9e34716a0d9bfca5b82d01e4782f11d5ba5cbc5d8dab6777f175ad81"} Jul 06 22:47:11.497068 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.497049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6cj5p" event={"ID":"74f6aec2-1b74-43cd-9a5d-840a409c64e0","Type":"ContainerStarted","Data":"a2e1699b4f1244e345ddd3298a1c71e7b6f711b52c7da30fc2df7bacaaf955c9"} Jul 06 22:47:11.498500 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.498469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nv2vk" event={"ID":"f4f73e02-0d1f-4c9e-bb6d-052c1a075e94","Type":"ContainerStarted","Data":"35ffe9fcb5d9ef43c9c74978a27860e2a456e82390fc7378a5ac4e6ca36b4ed3"} Jul 06 22:47:11.499792 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.499770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bh52b" event={"ID":"1db32628-6834-406d-afb6-eb4f71d1568c","Type":"ContainerStarted","Data":"01b97f781a43f050c91ae2e669ddf75f837003b7123b9d63abff52b38b10da42"} Jul 06 22:47:11.501076 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.501056 2579 generic.go:356] "Generic (PLEG): container finished" podID="a858fa8c-dc0c-4990-b370-513310efb011" containerID="eae34c472a9f306feba52ddd3cced8bd9552c56b337978ff1442d7a93b250a6c" exitCode=0 Jul 06 22:47:11.501176 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.501118 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerDied","Data":"eae34c472a9f306feba52ddd3cced8bd9552c56b337978ff1442d7a93b250a6c"} Jul 06 22:47:11.502610 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.502593 2579 generic.go:356] "Generic (PLEG): container finished" podID="dc46d6abc3ef28f9f3665a9927f901c6" containerID="6910dfe3831a9c8e63b008de0bd351fd2049972071b3415be3142a60fdb735ec" exitCode=0 Jul 06 22:47:11.502678 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.502665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" event={"ID":"dc46d6abc3ef28f9f3665a9927f901c6","Type":"ContainerDied","Data":"6910dfe3831a9c8e63b008de0bd351fd2049972071b3415be3142a60fdb735ec"} Jul 06 22:47:11.504158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.504135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" event={"ID":"aa1688f2-ab1e-4256-ab16-c93df1e71419","Type":"ContainerStarted","Data":"74f9ab4572d1236fb83b78a87454f951ce9ba9bab42b8adf53906980b064fe07"} Jul 06 22:47:11.508500 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.508464 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4wdrm" podStartSLOduration=8.902150719 podStartE2EDuration="23.508453859s" totalImagesPullingTime="14.60630314s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:11.508415318 +0000 UTC m=+23.644139872" watchObservedRunningTime="2026-07-06 22:47:11.508453859 +0000 UTC m=+23.644178413" Jul 06 22:47:11.508787 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.508758 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bxlzt" podStartSLOduration=4.009327368 podStartE2EDuration="23.50875078s" totalImagesPullingTime="19.499423412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:10.56341001 +0000 UTC m=+22.699134626" watchObservedRunningTime="2026-07-06 22:47:11.50875078 +0000 UTC m=+23.644475335" Jul 06 22:47:11.544306 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.544268 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bh52b" podStartSLOduration=3.38393989 podStartE2EDuration="22.544250445s" totalImagesPullingTime="19.160310555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:11.527407697 +0000 UTC m=+23.663132245" watchObservedRunningTime="2026-07-06 22:47:11.544250445 +0000 UTC m=+23.679975001" Jul 06 22:47:11.544406 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.544356 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6cj5p" podStartSLOduration=9.542207469000001 podStartE2EDuration="23.544352086s" totalImagesPullingTime="14.002144617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:11.543742212 +0000 UTC m=+23.679466767" watchObservedRunningTime="2026-07-06 22:47:11.544352086 +0000 UTC m=+23.680076641" Jul 06 22:47:11.605964 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.605908 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nv2vk" podStartSLOduration=8.986255612 podStartE2EDuration="23.605887364s" totalImagesPullingTime="14.619631752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:11.605313099 +0000 UTC m=+23.741037653" watchObservedRunningTime="2026-07-06 22:47:11.605887364 +0000 UTC m=+23.741611920" Jul 06 22:47:11.670598 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.670557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:11.744703 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:11.744621 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:12.239648 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.239558 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 22:47:12.313959 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.313930 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:47:12.319187 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.319054 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T22:47:12.239569484Z","UUID":"ee9abecc-7010-48be-8d92-7b6f3534c3eb","Handler":null,"Name":"","Endpoint":""} Jul 06 22:47:12.320667 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.320638 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 22:47:12.320772 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.320682 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 22:47:12.421946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.421866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:12.421946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.421908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:12.422272 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.421866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:12.422272 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:12.421981 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:12.422272 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:12.422067 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:12.422272 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:12.422172 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:12.508212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.508177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" event={"ID":"aa1688f2-ab1e-4256-ab16-c93df1e71419","Type":"ContainerStarted","Data":"27afcfac7cbd97fc06ba15eed223034e0db4cccd48a78937102bbc3cc213cb15"} Jul 06 22:47:12.510202 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.510172 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" event={"ID":"dc46d6abc3ef28f9f3665a9927f901c6","Type":"ContainerStarted","Data":"13fb512727a73cc17e4ad356a721d5899d49e87d0fc7b5a0169b1debd166fe94"} Jul 06 22:47:12.525841 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.525786 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-52.ec2.internal" podStartSLOduration=23.525767262 podStartE2EDuration="23.525767262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:12.525454655 +0000 UTC m=+24.661179210" watchObservedRunningTime="2026-07-06 22:47:12.525767262 +0000 UTC m=+24.661491819" Jul 06 22:47:12.685268 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.685195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:12.787984 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:12.787949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:13.516305 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:13.516279 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:47:13.516787 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:13.516732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"30eed2b27f3292fe656e4851feb76b6aa12beb54a6d1a1c85f100043965c3fe7"} Jul 06 22:47:13.573232 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:13.573197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:13.631353 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:13.631324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:13.959034 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:13.958949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.054807 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.054776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.155091 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.155059 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.255881 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.255806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.422420 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.422389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:14.422587 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.422389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:14.422587 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.422531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:14.422587 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:14.422561 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:14.422748 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:14.422636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:14.422812 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:14.422735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:14.521895 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.521817 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" event={"ID":"aa1688f2-ab1e-4256-ab16-c93df1e71419","Type":"ContainerStarted","Data":"a2e31c5d116ff8fa1a801971470858d7536419645515cf0ef734cbcfb3ecd5e3"} Jul 06 22:47:14.542308 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.542265 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8zk8" podStartSLOduration=5.154709079 podStartE2EDuration="26.542251398s" totalImagesPullingTime="21.387542319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:14.542003554 +0000 UTC m=+26.677728110" watchObservedRunningTime="2026-07-06 22:47:14.542251398 +0000 UTC m=+26.677975952" Jul 06 22:47:14.614365 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.614324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.688565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.688401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.903984 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.903911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:14.928887 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:14.928866 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wdrm_b507c59f-0773-4b0f-9d84-7e9aedb5561c/dns-node-resolver/0.log" Jul 06 22:47:15.003136 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:15.003094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:15.364307 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:15.364283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:47:15.364945 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:15.364929 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:47:15.525215 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:15.525184 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6cj5p" Jul 06 22:47:15.590170 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:15.590082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:15.928924 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:15.928861 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nv2vk_f4f73e02-0d1f-4c9e-bb6d-052c1a075e94/node-ca/0.log" Jul 06 22:47:16.401631 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.401455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:16.421915 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.421880 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:16.422060 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.421886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:16.422060 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:16.421991 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:16.422060 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.422005 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:16.422228 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:16.422144 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:16.422228 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:16.422199 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:16.490204 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.490092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:16.529916 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.529892 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:47:16.530567 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.530306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"edb516295774481a0a3a5d503a1a98482ebc7736feba81b72ea32df793f29a6c"} Jul 06 22:47:16.530567 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.530542 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:47:16.530801 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.530783 2579 scope.go:117] "RemoveContainer" containerID="a36a953d220352539d45bc2686a20e742cc13eacb2ef5a5dc184ddb2d0ad0ace" Jul 06 22:47:16.532390 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.532359 2579 generic.go:356] "Generic (PLEG): container finished" podID="a858fa8c-dc0c-4990-b370-513310efb011" containerID="f7cc5c6994430ed91e1a26d99784a1523d8d006e66a7ce5c2dce0e026e82ecb1" exitCode=0 Jul 06 22:47:16.532489 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.532438 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerDied","Data":"f7cc5c6994430ed91e1a26d99784a1523d8d006e66a7ce5c2dce0e026e82ecb1"} Jul 06 22:47:16.547022 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.547001 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:47:16.554656 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:16.554633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:17.479788 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:17.479753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:17.539897 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:17.539876 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:47:17.540333 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:17.540309 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" event={"ID":"e689f81a-bbf3-4c0b-8b1b-86b6c34f394f","Type":"ContainerStarted","Data":"6182b591d7c663abe4b3f4730d008bf4816437249265fcc4153f31d2e1472e08"} Jul 06 22:47:17.540865 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:17.540741 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:47:17.540865 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:17.540810 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:47:17.558382 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:17.558363 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:47:17.574615 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:17.574552 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" podStartSLOduration=8.953703647 podStartE2EDuration="28.574531623s" totalImagesPullingTime="19.620827976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:17.573826739 +0000 UTC m=+29.709551357" watchObservedRunningTime="2026-07-06 22:47:17.574531623 +0000 UTC m=+29.710256179" Jul 06 22:47:18.293833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.293793 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jmt2k"] Jul 06 22:47:18.294009 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.293945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:18.294083 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:18.294047 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:18.299282 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.299257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7ccm"] Jul 06 22:47:18.299400 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.299359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:18.299449 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:18.299432 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:18.302771 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.302750 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cmkgb"] Jul 06 22:47:18.302896 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.302858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:18.302979 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:18.302960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:18.545076 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.544990 2579 generic.go:356] "Generic (PLEG): container finished" podID="a858fa8c-dc0c-4990-b370-513310efb011" containerID="7b6ce6aa8c2ea70e4deb1d5d8038e297fd2b0c738f5f1f392c63e1b54203e751" exitCode=0 Jul 06 22:47:18.545455 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:18.545068 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerDied","Data":"7b6ce6aa8c2ea70e4deb1d5d8038e297fd2b0c738f5f1f392c63e1b54203e751"} Jul 06 22:47:20.143341 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.143177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:20.422187 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.422107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:20.422352 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.422103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:20.422352 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:20.422251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:20.422352 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.422290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:20.422473 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:20.422355 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:20.422473 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:20.422429 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:20.551690 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.551655 2579 generic.go:356] "Generic (PLEG): container finished" podID="a858fa8c-dc0c-4990-b370-513310efb011" containerID="3a97e166d9b6336ddd0bbcf69821d1806ed4d4d6240cdb43fa8ccdc4824b9515" exitCode=0 Jul 06 22:47:20.551833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.551703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerDied","Data":"3a97e166d9b6336ddd0bbcf69821d1806ed4d4d6240cdb43fa8ccdc4824b9515"} Jul 06 22:47:20.648029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.647996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:20.745803 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.745726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:20.847751 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.847723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:20.948313 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:20.948279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:21.175095 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:21.175009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:21.285285 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:21.285253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:21.385496 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:21.385461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:22.011244 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.011205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:22.011403 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.011383 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:47:22.011476 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.011462 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs podName:efadaccb-16ab-4bf9-aa7d-93db206c15ab nodeName:}" failed. No retries permitted until 2026-07-06 22:47:54.011439959 +0000 UTC m=+66.147164495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs") pod "network-metrics-daemon-cmkgb" (UID: "efadaccb-16ab-4bf9-aa7d-93db206c15ab") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:47:22.145315 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.145238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:22.212260 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.212228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:22.212712 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.212401 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:47:22.212712 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.212428 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:47:22.212712 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.212442 2579 projected.go:196] Error preparing data for projected volume kube-api-access-bspnl for pod openshift-network-diagnostics/network-check-target-jmt2k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:47:22.212712 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.212492 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl podName:2c329ad7-e344-4011-85b7-acb16c4fdb13 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:54.212479571 +0000 UTC m=+66.348204108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bspnl" (UniqueName: "kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl") pod "network-check-target-jmt2k" (UID: "2c329ad7-e344-4011-85b7-acb16c4fdb13") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:47:22.413787 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.413705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:22.413939 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.413819 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:47:22.413939 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.413876 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret podName:4a33cb94-f91f-4a25-85d5-850bd2ab71e3 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:54.413861448 +0000 UTC m=+66.549585981 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret") pod "global-pull-secret-syncer-v7ccm" (UID: "4a33cb94-f91f-4a25-85d5-850bd2ab71e3") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:47:22.421960 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.421931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:22.422102 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.421964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:22.422102 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.421934 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:22.422102 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.422059 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:22.422258 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.422174 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:22.422304 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:22.422263 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:22.537751 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.537716 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:22.941078 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:22.941043 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:23.337833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:23.337742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:23.540988 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:23.540944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:23.686982 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:23.686947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:23.937694 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:23.937596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:24.332854 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.332754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:24.422842 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.422814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:24.423351 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.422814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:24.423351 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.422964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:24.423351 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:24.422960 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:24.423351 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:24.423083 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:24.423351 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:24.423188 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:24.535857 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.535820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:24.563453 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.563424 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerStarted","Data":"43de18309be940b02f8d644e0addfc583f94a7d3e0d0326504f3092a51881924"} Jul 06 22:47:24.741546 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.741514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:24.942099 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:24.942069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:25.137805 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:25.137709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:25.198180 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:25.198149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:25.537589 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:25.537553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:25.567773 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:25.567747 2579 generic.go:356] "Generic (PLEG): container finished" podID="a858fa8c-dc0c-4990-b370-513310efb011" containerID="43de18309be940b02f8d644e0addfc583f94a7d3e0d0326504f3092a51881924" exitCode=0 Jul 06 22:47:25.567900 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:25.567799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerDied","Data":"43de18309be940b02f8d644e0addfc583f94a7d3e0d0326504f3092a51881924"} Jul 06 22:47:26.145794 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.145763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:26.422147 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.422051 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:26.422147 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.422096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:26.422147 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.422056 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:26.422415 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:26.422184 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:26.422415 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:26.422303 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:26.422415 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:26.422379 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:26.540801 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.540768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:26.573093 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.573062 2579 generic.go:356] "Generic (PLEG): container finished" podID="a858fa8c-dc0c-4990-b370-513310efb011" containerID="75bcb458f302714697d48769d3f836e1fe77c3b07f4b75019f28b7e8f6380610" exitCode=0 Jul 06 22:47:26.573265 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.573103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerDied","Data":"75bcb458f302714697d48769d3f836e1fe77c3b07f4b75019f28b7e8f6380610"} Jul 06 22:47:26.784743 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.784659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:26.886015 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.885984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:26.985502 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:26.985471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.337152 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:27.337106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.536348 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:27.536317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.578899 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:27.578865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-phs26" event={"ID":"a858fa8c-dc0c-4990-b370-513310efb011","Type":"ContainerStarted","Data":"ef04a8cfa8607b70083b84819e7ce591660a7d6568c5e9d8b749948630ebdb8e"} Jul 06 22:47:27.602290 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:27.602249 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-phs26" podStartSLOduration=11.101537221 podStartE2EDuration="38.602233472s" totalImagesPullingTime="27.500696251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:27.601110986 +0000 UTC m=+39.736835541" watchObservedRunningTime="2026-07-06 22:47:27.602233472 +0000 UTC m=+39.737958028" Jul 06 22:47:27.826361 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:27.826329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:27.879940 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:27.879860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:28.140892 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:28.140815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:28.422660 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:28.422635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:28.423952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:28.423148 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:28.423952 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:28.423013 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:28.425649 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:28.424187 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:28.425649 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:28.425159 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:28.425995 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:28.425971 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:28.736379 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:28.736250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:28.934543 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:28.934505 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:29.158686 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:29.158659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:29.339600 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:29.339571 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:30.340399 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:30.340365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:30.422072 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:30.422041 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:30.422072 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:30.422071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:30.422304 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:30.422052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:30.422304 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:30.422168 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:30.422304 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:30.422227 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:30.422304 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:30.422287 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:30.734930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:30.734896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:30.940182 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:30.940146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:31.142534 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:31.142512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:31.340335 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:31.340299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:32.421941 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:32.421904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:32.421941 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:32.421924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:32.422466 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:32.421914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:32.422466 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:32.422018 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:32.422466 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:32.422088 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:32.422466 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:32.422172 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:34.425391 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:34.425360 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:34.425865 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:34.425456 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:34.425865 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:34.425477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:34.425865 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:34.425458 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:34.425865 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:34.425570 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:34.425865 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:34.425603 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:36.422532 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:36.422492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:36.423002 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:36.422489 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:36.423002 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:36.422610 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cmkgb" podUID="efadaccb-16ab-4bf9-aa7d-93db206c15ab" Jul 06 22:47:36.423002 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:36.422707 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jmt2k" podUID="2c329ad7-e344-4011-85b7-acb16c4fdb13" Jul 06 22:47:36.423002 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:36.422501 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:36.423002 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:36.422811 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v7ccm" podUID="4a33cb94-f91f-4a25-85d5-850bd2ab71e3" Jul 06 22:47:36.659640 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:36.659604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:36.760185 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:36.760079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:36.859381 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:36.859346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:36.958051 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:36.958022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:37.168194 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.168169 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-52.ec2.internal" event="NodeReady" Jul 06 22:47:37.168368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.168292 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 22:47:37.219488 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.219454 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c2smb"] Jul 06 22:47:37.245179 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.245153 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5gc9m"] Jul 06 22:47:37.245331 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.245263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.248428 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.248402 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ctk6j\"" Jul 06 22:47:37.248565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.248434 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 22:47:37.248565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.248484 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 22:47:37.269166 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.269141 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2smb"] Jul 06 22:47:37.269295 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.269178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gc9m"] Jul 06 22:47:37.269295 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.269194 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p8q4d"] Jul 06 22:47:37.269295 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.269243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.271518 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.271501 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 22:47:37.271904 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.271884 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 22:47:37.272003 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.271912 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c4v9t\"" Jul 06 22:47:37.272003 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.271970 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 22:47:37.288821 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.288794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p8q4d"] Jul 06 22:47:37.288908 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.288891 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.291660 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.291643 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 22:47:37.291733 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.291698 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 22:47:37.291799 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.291780 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xdg5g\"" Jul 06 22:47:37.291859 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.291782 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 22:47:37.292599 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.292584 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 22:47:37.385777 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.385713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:37.424922 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.424847 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93834e1-fa53-48d3-a879-b35a940decac-data-volume\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.424922 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.424901 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acceb5a1-7065-409a-a770-4787d1a7642a-metrics-tls\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.424925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f93834e1-fa53-48d3-a879-b35a940decac-crio-socket\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.424949 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k659s\" (UniqueName: \"kubernetes.io/projected/f93834e1-fa53-48d3-a879-b35a940decac-kube-api-access-k659s\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.424979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f93834e1-fa53-48d3-a879-b35a940decac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.424995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f93834e1-fa53-48d3-a879-b35a940decac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.425010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acceb5a1-7065-409a-a770-4787d1a7642a-config-volume\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.425065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee-cert\") pod \"ingress-canary-5gc9m\" (UID: \"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee\") " pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.425097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acceb5a1-7065-409a-a770-4787d1a7642a-tmp-dir\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.425115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwv7\" (UniqueName: \"kubernetes.io/projected/acceb5a1-7065-409a-a770-4787d1a7642a-kube-api-access-dtwv7\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.425303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.425214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnt5v\" (UniqueName: \"kubernetes.io/projected/f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee-kube-api-access-fnt5v\") pod \"ingress-canary-5gc9m\" (UID: \"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee\") " pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.490519 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.490485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:37.525495 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f93834e1-fa53-48d3-a879-b35a940decac-crio-socket\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.525628 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k659s\" (UniqueName: \"kubernetes.io/projected/f93834e1-fa53-48d3-a879-b35a940decac-kube-api-access-k659s\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.525628 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f93834e1-fa53-48d3-a879-b35a940decac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.525628 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f93834e1-fa53-48d3-a879-b35a940decac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.525628 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acceb5a1-7065-409a-a770-4787d1a7642a-config-volume\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.525817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee-cert\") pod \"ingress-canary-5gc9m\" (UID: \"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee\") " pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.525817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acceb5a1-7065-409a-a770-4787d1a7642a-tmp-dir\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.525817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwv7\" (UniqueName: \"kubernetes.io/projected/acceb5a1-7065-409a-a770-4787d1a7642a-kube-api-access-dtwv7\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.525817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f93834e1-fa53-48d3-a879-b35a940decac-crio-socket\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.526043 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnt5v\" (UniqueName: \"kubernetes.io/projected/f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee-kube-api-access-fnt5v\") pod \"ingress-canary-5gc9m\" (UID: \"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee\") " pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.526043 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93834e1-fa53-48d3-a879-b35a940decac-data-volume\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.526043 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.525908 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acceb5a1-7065-409a-a770-4787d1a7642a-metrics-tls\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.526043 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.526029 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acceb5a1-7065-409a-a770-4787d1a7642a-tmp-dir\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.526273 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.526249 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f93834e1-fa53-48d3-a879-b35a940decac-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.526337 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.526294 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f93834e1-fa53-48d3-a879-b35a940decac-data-volume\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.526391 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.526344 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/acceb5a1-7065-409a-a770-4787d1a7642a-config-volume\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.529844 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.529814 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/acceb5a1-7065-409a-a770-4787d1a7642a-metrics-tls\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.529949 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.529814 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f93834e1-fa53-48d3-a879-b35a940decac-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.529949 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.529818 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee-cert\") pod \"ingress-canary-5gc9m\" (UID: \"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee\") " pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.541836 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.541810 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k659s\" (UniqueName: \"kubernetes.io/projected/f93834e1-fa53-48d3-a879-b35a940decac-kube-api-access-k659s\") pod \"insights-runtime-extractor-p8q4d\" (UID: \"f93834e1-fa53-48d3-a879-b35a940decac\") " pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.551545 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.551518 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwv7\" (UniqueName: \"kubernetes.io/projected/acceb5a1-7065-409a-a770-4787d1a7642a-kube-api-access-dtwv7\") pod \"dns-default-c2smb\" (UID: \"acceb5a1-7065-409a-a770-4787d1a7642a\") " pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.553364 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.553336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnt5v\" (UniqueName: \"kubernetes.io/projected/f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee-kube-api-access-fnt5v\") pod \"ingress-canary-5gc9m\" (UID: \"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee\") " pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.555194 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.555176 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:37.577790 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.577763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gc9m" Jul 06 22:47:37.596256 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.596219 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p8q4d" Jul 06 22:47:37.771640 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.771609 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c2smb"] Jul 06 22:47:37.774692 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:37.774658 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacceb5a1_7065_409a_a770_4787d1a7642a.slice/crio-4eb94306116413c57daa4b0b15df60b8815afa9632c20f795daf9d49b4baf0d7 WatchSource:0}: Error finding container 4eb94306116413c57daa4b0b15df60b8815afa9632c20f795daf9d49b4baf0d7: Status 404 returned error can't find the container with id 4eb94306116413c57daa4b0b15df60b8815afa9632c20f795daf9d49b4baf0d7 Jul 06 22:47:37.788284 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.788259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gc9m"] Jul 06 22:47:37.791525 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:37.791494 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cf9a80_4a40_43a7_bb15_c66d2f7af8ee.slice/crio-a65a382f3cdd5b132d8ce637ffde9cf61dbe81a6a0b897c06e674c22a261f84a WatchSource:0}: Error finding container a65a382f3cdd5b132d8ce637ffde9cf61dbe81a6a0b897c06e674c22a261f84a: Status 404 returned error can't find the container with id a65a382f3cdd5b132d8ce637ffde9cf61dbe81a6a0b897c06e674c22a261f84a Jul 06 22:47:37.807847 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:37.807822 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p8q4d"] Jul 06 22:47:37.811147 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:37.811111 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93834e1_fa53_48d3_a879_b35a940decac.slice/crio-f1fcf21d9d83d2782743ea6af2c959bfb42673652e168c4f8f3cb431543870bc WatchSource:0}: Error finding container f1fcf21d9d83d2782743ea6af2c959bfb42673652e168c4f8f3cb431543870bc: Status 404 returned error can't find the container with id f1fcf21d9d83d2782743ea6af2c959bfb42673652e168c4f8f3cb431543870bc Jul 06 22:47:38.422546 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.422484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:38.423989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.423954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:38.424913 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.424628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:38.426508 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.426482 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:47:38.427707 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.427685 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9ddx\"" Jul 06 22:47:38.427939 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.427922 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:47:38.428180 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.428164 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vmht5\"" Jul 06 22:47:38.428294 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.428276 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:47:38.428394 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.428378 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:47:38.605075 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.605037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gc9m" event={"ID":"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee","Type":"ContainerStarted","Data":"a65a382f3cdd5b132d8ce637ffde9cf61dbe81a6a0b897c06e674c22a261f84a"} Jul 06 22:47:38.606998 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.606965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8q4d" event={"ID":"f93834e1-fa53-48d3-a879-b35a940decac","Type":"ContainerStarted","Data":"f1fcf21d9d83d2782743ea6af2c959bfb42673652e168c4f8f3cb431543870bc"} Jul 06 22:47:38.608960 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:38.608892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2smb" event={"ID":"acceb5a1-7065-409a-a770-4787d1a7642a","Type":"ContainerStarted","Data":"4eb94306116413c57daa4b0b15df60b8815afa9632c20f795daf9d49b4baf0d7"} Jul 06 22:47:39.560786 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:39.560733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:39.613629 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:39.613595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:40.844197 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:40.844151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:40.901490 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:40.901459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:44.603603 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.603553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:44.627958 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.627535 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gc9m" event={"ID":"f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee","Type":"ContainerStarted","Data":"84f5054d8529e6ee00adc1971022af96f7c5c5540ccab9db9cf61647f84d5d1c"} Jul 06 22:47:44.629500 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.629469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8q4d" event={"ID":"f93834e1-fa53-48d3-a879-b35a940decac","Type":"ContainerStarted","Data":"2056b2f903eb16968d004190198ef83a04e7911c66caa2129c17402bc34f2367"} Jul 06 22:47:44.631576 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.631551 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2smb" event={"ID":"acceb5a1-7065-409a-a770-4787d1a7642a","Type":"ContainerStarted","Data":"384ca0da38f96c2d987152e0ab44e992b9535c80daac91c33b314168088a7359"} Jul 06 22:47:44.631737 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.631594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c2smb" event={"ID":"acceb5a1-7065-409a-a770-4787d1a7642a","Type":"ContainerStarted","Data":"c123461cb8073ff1cc09c9d9ed8a91066f6fe26584cf33f5a9194f9650f59cc2"} Jul 06 22:47:44.631737 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.631688 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:44.641880 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.641841 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5gc9m" podStartSLOduration=1.6673695899999998 podStartE2EDuration="7.6418291s" totalImagesPullingTime="5.97445951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:44.641288368 +0000 UTC m=+56.777012922" watchObservedRunningTime="2026-07-06 22:47:44.6418291 +0000 UTC m=+56.777553656" Jul 06 22:47:44.656188 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.656146 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-c2smb" podStartSLOduration=1.664885331 podStartE2EDuration="7.656108103s" totalImagesPullingTime="5.991222772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:44.655682492 +0000 UTC m=+56.791407049" watchObservedRunningTime="2026-07-06 22:47:44.656108103 +0000 UTC m=+56.791832660" Jul 06 22:47:44.701614 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.701591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:44.763235 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.763204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:44.765799 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.765779 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh"] Jul 06 22:47:44.767405 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.767392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" Jul 06 22:47:44.769473 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.769452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 22:47:44.769558 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.769501 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-57bqg\"" Jul 06 22:47:44.776561 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.776533 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh"] Jul 06 22:47:44.788240 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.788207 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cd34a40-7027-4e9e-927d-1d5ba2916c06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-4vvdh\" (UID: \"6cd34a40-7027-4e9e-927d-1d5ba2916c06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" Jul 06 22:47:44.888967 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.888885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cd34a40-7027-4e9e-927d-1d5ba2916c06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-4vvdh\" (UID: \"6cd34a40-7027-4e9e-927d-1d5ba2916c06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" Jul 06 22:47:44.891198 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.891174 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6cd34a40-7027-4e9e-927d-1d5ba2916c06-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-4vvdh\" (UID: \"6cd34a40-7027-4e9e-927d-1d5ba2916c06\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" Jul 06 22:47:44.911267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:44.911244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:45.075574 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:45.075525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" Jul 06 22:47:45.198558 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:45.198528 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh"] Jul 06 22:47:45.201903 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:45.201878 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd34a40_7027_4e9e_927d_1d5ba2916c06.slice/crio-2c068f86343f633d0314d2929b471059b23a0ecbd0139c6f860487b3a3a7b71d WatchSource:0}: Error finding container 2c068f86343f633d0314d2929b471059b23a0ecbd0139c6f860487b3a3a7b71d: Status 404 returned error can't find the container with id 2c068f86343f633d0314d2929b471059b23a0ecbd0139c6f860487b3a3a7b71d Jul 06 22:47:45.637702 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:45.637614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8q4d" event={"ID":"f93834e1-fa53-48d3-a879-b35a940decac","Type":"ContainerStarted","Data":"666cf8ae16245e4d0e55cb4dff833352c2f95c543bf3f924c6023f37c713c01e"} Jul 06 22:47:45.638899 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:45.638869 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" event={"ID":"6cd34a40-7027-4e9e-927d-1d5ba2916c06","Type":"ContainerStarted","Data":"2c068f86343f633d0314d2929b471059b23a0ecbd0139c6f860487b3a3a7b71d"} Jul 06 22:47:46.644351 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.644321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" event={"ID":"6cd34a40-7027-4e9e-927d-1d5ba2916c06","Type":"ContainerStarted","Data":"95d8db1319662a02375a4b25edb83c5a0c3d73a8e68f9c7cdec607f88b17d124"} Jul 06 22:47:46.644717 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.644541 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" Jul 06 22:47:46.649913 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.649885 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" Jul 06 22:47:46.660650 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.660600 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-4vvdh" podStartSLOduration=1.6219944229999999 podStartE2EDuration="2.660580206s" totalImagesPullingTime="1.038585783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:46.65971 +0000 UTC m=+58.795434556" watchObservedRunningTime="2026-07-06 22:47:46.660580206 +0000 UTC m=+58.796304763" Jul 06 22:47:46.792884 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.792849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:46.831784 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.831757 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw"] Jul 06 22:47:46.836621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.836601 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:46.838989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.838776 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 22:47:46.838989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.838813 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 22:47:46.838989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.838853 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 22:47:46.838989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.838875 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 22:47:46.838989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.838891 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wjzvb\"" Jul 06 22:47:46.838989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.838860 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 22:47:46.842158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.842136 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw"] Jul 06 22:47:46.863409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.863381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:46.905772 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.905697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:46.905772 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.905764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64b4\" (UniqueName: \"kubernetes.io/projected/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-kube-api-access-m64b4\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:46.905997 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.905824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:46.905997 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:46.905855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.007028 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.006991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.007221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.007037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.007221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.007087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64b4\" (UniqueName: \"kubernetes.io/projected/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-kube-api-access-m64b4\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.007221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.007175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.007221 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:47.007186 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 06 22:47:47.007534 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:47.007261 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-tls podName:81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d nodeName:}" failed. No retries permitted until 2026-07-06 22:47:47.507241548 +0000 UTC m=+59.642966095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-dp9qw" (UID: "81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d") : secret "prometheus-operator-tls" not found Jul 06 22:47:47.008012 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.007978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.009809 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.009786 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.018035 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.018016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64b4\" (UniqueName: \"kubernetes.io/projected/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-kube-api-access-m64b4\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.050574 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.050545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:47.510679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.510647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.512799 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.512780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-dp9qw\" (UID: \"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.649440 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.649410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p8q4d" event={"ID":"f93834e1-fa53-48d3-a879-b35a940decac","Type":"ContainerStarted","Data":"f980c74b357ad06ce5484b566977b01e10b4fc6b6a4dc54ee0cef4f417db4008"} Jul 06 22:47:47.666029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.665671 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p8q4d" podStartSLOduration=1.274561619 podStartE2EDuration="10.665654385s" totalImagesPullingTime="9.391092766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:47:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:47.665267725 +0000 UTC m=+59.800992305" watchObservedRunningTime="2026-07-06 22:47:47.665654385 +0000 UTC m=+59.801378941" Jul 06 22:47:47.746976 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.746949 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" Jul 06 22:47:47.861835 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:47.861808 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw"] Jul 06 22:47:47.864928 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:47.864900 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aa2d02_f9e8_4c0a_b1cc_4c1f0204730d.slice/crio-acf1a4f46816b9d4fc01ac370e2638ef6b1fad1f0d61d109952c1352100af942 WatchSource:0}: Error finding container acf1a4f46816b9d4fc01ac370e2638ef6b1fad1f0d61d109952c1352100af942: Status 404 returned error can't find the container with id acf1a4f46816b9d4fc01ac370e2638ef6b1fad1f0d61d109952c1352100af942 Jul 06 22:47:48.654383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:48.654345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" event={"ID":"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d","Type":"ContainerStarted","Data":"acf1a4f46816b9d4fc01ac370e2638ef6b1fad1f0d61d109952c1352100af942"} Jul 06 22:47:49.246040 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:49.246008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:49.351215 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:49.351182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:49.561788 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:49.561714 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw68r" Jul 06 22:47:49.670667 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:49.670631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:49.748238 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:49.748199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:49.808009 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:49.807953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:49.908661 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:49.908616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:50.470119 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:50.470063 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:50.568830 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:50.568795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:50.663364 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:50.663328 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" event={"ID":"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d","Type":"ContainerStarted","Data":"b3ac121bfbf63998e17acfd7696298b7a98ce3c2fd98dcb9ec481aab3de0892a"} Jul 06 22:47:50.663529 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:50.663390 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" event={"ID":"81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d","Type":"ContainerStarted","Data":"598c1cbfe314036bee213d33c59075a0fd7d95899e56a40cb1f7a2a83254b41c"} Jul 06 22:47:50.678915 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:50.678868 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-dp9qw" podStartSLOduration=2.982987698 podStartE2EDuration="4.678852808s" totalImagesPullingTime="1.69586511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:50.678119219 +0000 UTC m=+62.813843811" watchObservedRunningTime="2026-07-06 22:47:50.678852808 +0000 UTC m=+62.814577384" Jul 06 22:47:51.929677 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:51.929644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.192890 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.192795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.223765 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.223733 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg"] Jul 06 22:47:52.226298 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.226273 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.228567 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.228536 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq"] Jul 06 22:47:52.228679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.228663 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 22:47:52.228742 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.228675 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 22:47:52.228951 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.228932 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:47:52.229047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.228955 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-klr96\"" Jul 06 22:47:52.230562 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.230548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.232997 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.232978 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gqm8g\"" Jul 06 22:47:52.233389 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.233311 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 22:47:52.233594 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.233544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 22:47:52.241342 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.241325 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg"] Jul 06 22:47:52.246091 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.246069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq"] Jul 06 22:47:52.260626 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.260604 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n2cnq"] Jul 06 22:47:52.262575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.262556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.266591 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.266570 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 22:47:52.266887 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.266874 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 22:47:52.267596 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.267579 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 22:47:52.267917 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.267900 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njcl6\"" Jul 06 22:47:52.296584 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.296560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.345717 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.345842 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwkmb\" (UniqueName: \"kubernetes.io/projected/0b90bb90-5d01-4b71-8025-61a1ed6627fd-kube-api-access-xwkmb\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.345842 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345746 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcp27\" (UniqueName: \"kubernetes.io/projected/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-api-access-rcp27\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.345842 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-textfile\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.345971 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.345971 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0751e64e-0b6c-4253-b6c3-879f09840d04-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.345971 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhslc\" (UniqueName: \"kubernetes.io/projected/0751e64e-0b6c-4253-b6c3-879f09840d04-kube-api-access-nhslc\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.345971 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-wtmp\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.345971 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b90bb90-5d01-4b71-8025-61a1ed6627fd-metrics-client-ca\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.345976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346073 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-root\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-sys\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.346221 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.346443 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.346443 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.346256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.446763 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.446763 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.446763 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.446837 2579 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.446889 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.446898 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-tls podName:0751e64e-0b6c-4253-b6c3-879f09840d04 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:52.946879055 +0000 UTC m=+65.082603590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-rpszq" (UID: "0751e64e-0b6c-4253-b6c3-879f09840d04") : secret "openshift-state-metrics-tls" not found Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.446924 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-tls podName:8aa77dd4-6ea9-4e8d-aea9-a385229b30ab nodeName:}" failed. No retries permitted until 2026-07-06 22:47:52.946913113 +0000 UTC m=+65.082637655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-gntjg" (UID: "8aa77dd4-6ea9-4e8d-aea9-a385229b30ab") : secret "kube-state-metrics-tls" not found Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwkmb\" (UniqueName: \"kubernetes.io/projected/0b90bb90-5d01-4b71-8025-61a1ed6627fd-kube-api-access-xwkmb\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447010 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.446996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcp27\" (UniqueName: \"kubernetes.io/projected/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-api-access-rcp27\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-textfile\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0751e64e-0b6c-4253-b6c3-879f09840d04-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhslc\" (UniqueName: \"kubernetes.io/projected/0751e64e-0b6c-4253-b6c3-879f09840d04-kube-api-access-nhslc\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-wtmp\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b90bb90-5d01-4b71-8025-61a1ed6627fd-metrics-client-ca\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-root\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-sys\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.447468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-textfile\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.448032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.448032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.447838 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b90bb90-5d01-4b71-8025-61a1ed6627fd-metrics-client-ca\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.448156 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.448032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.448156 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.448045 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.448156 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.448079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-root\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.448156 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.448138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-sys\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.448369 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.448165 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 22:47:52.448369 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.448197 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-wtmp\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.448369 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.448224 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls podName:0b90bb90-5d01-4b71-8025-61a1ed6627fd nodeName:}" failed. No retries permitted until 2026-07-06 22:47:52.94820534 +0000 UTC m=+65.083929890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls") pod "node-exporter-n2cnq" (UID: "0b90bb90-5d01-4b71-8025-61a1ed6627fd") : secret "node-exporter-tls" not found Jul 06 22:47:52.448764 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.448740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.448858 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.448841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0751e64e-0b6c-4253-b6c3-879f09840d04-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.449773 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.449747 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.449932 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.449914 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.450823 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.450802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.456500 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.456479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwkmb\" (UniqueName: \"kubernetes.io/projected/0b90bb90-5d01-4b71-8025-61a1ed6627fd-kube-api-access-xwkmb\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.457324 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.457303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhslc\" (UniqueName: \"kubernetes.io/projected/0751e64e-0b6c-4253-b6c3-879f09840d04-kube-api-access-nhslc\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.457789 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.457769 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcp27\" (UniqueName: \"kubernetes.io/projected/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-api-access-rcp27\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.473525 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.473502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.631968 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.631937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.767231 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.767146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.887275 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.887248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:52.950088 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.950057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:52.950565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.950095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.950565 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.950137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.950565 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.950224 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 06 22:47:52.950565 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:52.950294 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls podName:0b90bb90-5d01-4b71-8025-61a1ed6627fd nodeName:}" failed. No retries permitted until 2026-07-06 22:47:53.950280013 +0000 UTC m=+66.086004549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls") pod "node-exporter-n2cnq" (UID: "0b90bb90-5d01-4b71-8025-61a1ed6627fd") : secret "node-exporter-tls" not found Jul 06 22:47:52.952598 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.952574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0751e64e-0b6c-4253-b6c3-879f09840d04-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-rpszq\" (UID: \"0751e64e-0b6c-4253-b6c3-879f09840d04\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:52.952598 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.952589 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8aa77dd4-6ea9-4e8d-aea9-a385229b30ab-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-gntjg\" (UID: \"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:52.987006 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:52.986982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:53.137065 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.136986 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" Jul 06 22:47:53.142385 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.142361 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" Jul 06 22:47:53.220884 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.220840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:53.283896 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.283874 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg"] Jul 06 22:47:53.285776 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:53.285752 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa77dd4_6ea9_4e8d_aea9_a385229b30ab.slice/crio-bbe182739be3a8bf343028d3e2bbe2b9c7435a29406fc3e57c9ba0017f551f78 WatchSource:0}: Error finding container bbe182739be3a8bf343028d3e2bbe2b9c7435a29406fc3e57c9ba0017f551f78: Status 404 returned error can't find the container with id bbe182739be3a8bf343028d3e2bbe2b9c7435a29406fc3e57c9ba0017f551f78 Jul 06 22:47:53.314409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.314378 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq"] Jul 06 22:47:53.317394 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:53.317369 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0751e64e_0b6c_4253_b6c3_879f09840d04.slice/crio-6cd94d85d2bad525ab7c19ac228edb902204f4446d04e9109f9c8adb24f6f4f1 WatchSource:0}: Error finding container 6cd94d85d2bad525ab7c19ac228edb902204f4446d04e9109f9c8adb24f6f4f1: Status 404 returned error can't find the container with id 6cd94d85d2bad525ab7c19ac228edb902204f4446d04e9109f9c8adb24f6f4f1 Jul 06 22:47:53.360095 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.360073 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:53.395239 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.395219 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:47:53.398218 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.398193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.401199 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.401177 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 22:47:53.401301 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.401286 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 22:47:53.401528 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.401514 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ds5zr\"" Jul 06 22:47:53.401733 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.401649 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 22:47:53.401733 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.401700 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 22:47:53.401733 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.401701 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 22:47:53.401929 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.401765 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 22:47:53.402236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.402214 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 22:47:53.402339 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.402259 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 22:47:53.402339 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.402271 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 22:47:53.415947 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.415926 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:47:53.454886 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.454854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455025 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.454902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455025 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-web-config\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455119 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455119 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455119 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455096 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99b9d\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-kube-api-access-99b9d\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455255 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455255 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455333 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455373 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455406 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-out\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455435 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.455472 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.455444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.461205 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.461164 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:53.556590 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.556590 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.556810 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556630 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.556810 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.556810 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-out\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.556810 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.556810 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557061 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:53.556829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle podName:6a48657d-68bf-4f4f-bef0-e187cc0f6c62 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:54.056808807 +0000 UTC m=+66.192533359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62") : configmap references non-existent config key: ca-bundle.crt Jul 06 22:47:53.557061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-web-config\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.556994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.557022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.557023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557427 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.557087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99b9d\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-kube-api-access-99b9d\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.557612 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.557590 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.559753 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.559726 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:53.559861 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.559791 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-out\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.559934 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.559875 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.560533 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.560491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.560628 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.560578 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.560691 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.560671 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-volume\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.560748 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.560728 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.560992 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.560971 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.561225 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.561205 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.562421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.562403 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-web-config\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.569902 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.569881 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99b9d\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-kube-api-access-99b9d\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:53.674589 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.674509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" event={"ID":"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab","Type":"ContainerStarted","Data":"bbe182739be3a8bf343028d3e2bbe2b9c7435a29406fc3e57c9ba0017f551f78"} Jul 06 22:47:53.676253 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.676210 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" event={"ID":"0751e64e-0b6c-4253-b6c3-879f09840d04","Type":"ContainerStarted","Data":"9358e27ca8863b289ccf2f3cad49448e99c19d0cc889da0a2f9d3fe8767f80d4"} Jul 06 22:47:53.676363 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.676268 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" event={"ID":"0751e64e-0b6c-4253-b6c3-879f09840d04","Type":"ContainerStarted","Data":"afa523cf15f5bd953a3506e11182a0c5472b5a419cb36a9313d05a26f06e4526"} Jul 06 22:47:53.676363 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.676287 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" event={"ID":"0751e64e-0b6c-4253-b6c3-879f09840d04","Type":"ContainerStarted","Data":"6cd94d85d2bad525ab7c19ac228edb902204f4446d04e9109f9c8adb24f6f4f1"} Jul 06 22:47:53.960522 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.960434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:53.963461 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:53.963421 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0b90bb90-5d01-4b71-8025-61a1ed6627fd-node-exporter-tls\") pod \"node-exporter-n2cnq\" (UID: \"0b90bb90-5d01-4b71-8025-61a1ed6627fd\") " pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:54.061046 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.061010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:54.061258 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.061063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:54.062063 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.062033 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:54.063522 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.063497 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:47:54.071643 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.071614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n2cnq" Jul 06 22:47:54.074280 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.074260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efadaccb-16ab-4bf9-aa7d-93db206c15ab-metrics-certs\") pod \"network-metrics-daemon-cmkgb\" (UID: \"efadaccb-16ab-4bf9-aa7d-93db206c15ab\") " pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:54.199180 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.199143 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-77556fbd49-v8jdm"] Jul 06 22:47:54.202457 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.202431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.204873 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.204831 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 22:47:54.205019 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.204895 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 22:47:54.205019 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.204906 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 22:47:54.205469 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.205438 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-279gpg6osqfm0\"" Jul 06 22:47:54.205549 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.205480 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 22:47:54.205549 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.205497 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 22:47:54.209414 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.206374 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2zdp4\"" Jul 06 22:47:54.214442 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.214419 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77556fbd49-v8jdm"] Jul 06 22:47:54.262694 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.262827 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.262827 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qchjl\" (UniqueName: \"kubernetes.io/projected/69d471a2-071d-4716-9402-6bda4d881c6f-kube-api-access-qchjl\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.262827 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:54.262981 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-tls\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.262981 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.262981 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.262981 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.262964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-grpc-tls\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.263176 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.263010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69d471a2-071d-4716-9402-6bda4d881c6f-metrics-client-ca\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.265264 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.265240 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:47:54.275928 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.275910 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:47:54.286781 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.286760 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bspnl\" (UniqueName: \"kubernetes.io/projected/2c329ad7-e344-4011-85b7-acb16c4fdb13-kube-api-access-bspnl\") pod \"network-check-target-jmt2k\" (UID: \"2c329ad7-e344-4011-85b7-acb16c4fdb13\") " pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:54.322576 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.322548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:47:54.349664 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.349643 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vmht5\"" Jul 06 22:47:54.357384 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.357364 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cmkgb" Jul 06 22:47:54.357652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.357635 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9ddx\"" Jul 06 22:47:54.363844 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.363821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.363938 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.363863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qchjl\" (UniqueName: \"kubernetes.io/projected/69d471a2-071d-4716-9402-6bda4d881c6f-kube-api-access-qchjl\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.363938 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.363904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-tls\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.364041 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.363934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.364101 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.364077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.364165 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.364145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-grpc-tls\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.364219 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.364206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69d471a2-071d-4716-9402-6bda4d881c6f-metrics-client-ca\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.364451 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.364429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.365032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.364981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69d471a2-071d-4716-9402-6bda4d881c6f-metrics-client-ca\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.365862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.365831 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:47:54.366752 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.366729 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.366848 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.366827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.367158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.367099 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-grpc-tls\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.367158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.367149 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.367290 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.367195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.367326 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.367284 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69d471a2-071d-4716-9402-6bda4d881c6f-secret-thanos-querier-tls\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.378232 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.378212 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qchjl\" (UniqueName: \"kubernetes.io/projected/69d471a2-071d-4716-9402-6bda4d881c6f-kube-api-access-qchjl\") pod \"thanos-querier-77556fbd49-v8jdm\" (UID: \"69d471a2-071d-4716-9402-6bda4d881c6f\") " pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.465042 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.464956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:54.467635 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.467612 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:47:54.478204 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.478179 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a33cb94-f91f-4a25-85d5-850bd2ab71e3-original-pull-secret\") pod \"global-pull-secret-syncer-v7ccm\" (UID: \"4a33cb94-f91f-4a25-85d5-850bd2ab71e3\") " pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:54.517236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.517184 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:47:54.638860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.638836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v7ccm" Jul 06 22:47:54.641648 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.641620 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c2smb" Jul 06 22:47:54.666955 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:54.666933 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b90bb90_5d01_4b71_8025_61a1ed6627fd.slice/crio-caba2be0b7b44c153fda8dc47dd60a57dfa1848a6ea29a73e6409593bfad5b2e WatchSource:0}: Error finding container caba2be0b7b44c153fda8dc47dd60a57dfa1848a6ea29a73e6409593bfad5b2e: Status 404 returned error can't find the container with id caba2be0b7b44c153fda8dc47dd60a57dfa1848a6ea29a73e6409593bfad5b2e Jul 06 22:47:54.681443 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.681416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cnq" event={"ID":"0b90bb90-5d01-4b71-8025-61a1ed6627fd","Type":"ContainerStarted","Data":"caba2be0b7b44c153fda8dc47dd60a57dfa1848a6ea29a73e6409593bfad5b2e"} Jul 06 22:47:54.826470 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.822438 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-4hzkn"] Jul 06 22:47:54.828009 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.827981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-4hzkn" Jul 06 22:47:54.836146 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.834449 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6sq42\"" Jul 06 22:47:54.836146 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.834687 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 22:47:54.836146 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.834861 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 22:47:54.837855 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.837796 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-4hzkn"] Jul 06 22:47:54.844467 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.844428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:54.869040 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.868464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/bc362fc8-5494-472c-9b01-122d414ab850-kube-api-access-hk447\") pod \"downloads-685bfb5f64-4hzkn\" (UID: \"bc362fc8-5494-472c-9b01-122d414ab850\") " pod="openshift-console/downloads-685bfb5f64-4hzkn" Jul 06 22:47:54.875370 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.875156 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cmkgb"] Jul 06 22:47:54.892564 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:54.892529 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefadaccb_16ab_4bf9_aa7d_93db206c15ab.slice/crio-8fe88377e6e96e4576ff19d9276d41c35c94bef62bb92439c17f840239837a03 WatchSource:0}: Error finding container 8fe88377e6e96e4576ff19d9276d41c35c94bef62bb92439c17f840239837a03: Status 404 returned error can't find the container with id 8fe88377e6e96e4576ff19d9276d41c35c94bef62bb92439c17f840239837a03 Jul 06 22:47:54.941610 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.941566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:54.969151 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.969066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/bc362fc8-5494-472c-9b01-122d414ab850-kube-api-access-hk447\") pod \"downloads-685bfb5f64-4hzkn\" (UID: \"bc362fc8-5494-472c-9b01-122d414ab850\") " pod="openshift-console/downloads-685bfb5f64-4hzkn" Jul 06 22:47:54.981692 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:54.981654 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk447\" (UniqueName: \"kubernetes.io/projected/bc362fc8-5494-472c-9b01-122d414ab850-kube-api-access-hk447\") pod \"downloads-685bfb5f64-4hzkn\" (UID: \"bc362fc8-5494-472c-9b01-122d414ab850\") " pod="openshift-console/downloads-685bfb5f64-4hzkn" Jul 06 22:47:55.042405 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.042336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:55.108372 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.108340 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v7ccm"] Jul 06 22:47:55.112163 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:55.112114 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a33cb94_f91f_4a25_85d5_850bd2ab71e3.slice/crio-9bcaa1faa5cfc980fe58d6c413c0355fe21c13ccdbd0bb68d11f299a00449555 WatchSource:0}: Error finding container 9bcaa1faa5cfc980fe58d6c413c0355fe21c13ccdbd0bb68d11f299a00449555: Status 404 returned error can't find the container with id 9bcaa1faa5cfc980fe58d6c413c0355fe21c13ccdbd0bb68d11f299a00449555 Jul 06 22:47:55.116975 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.116740 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jmt2k"] Jul 06 22:47:55.119142 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:55.119086 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a48657d_68bf_4f4f_bef0_e187cc0f6c62.slice/crio-0ff3994b9bf9dad884e6f6749eefcbef73046610dd42ab7eb19d5c971bd4659f WatchSource:0}: Error finding container 0ff3994b9bf9dad884e6f6749eefcbef73046610dd42ab7eb19d5c971bd4659f: Status 404 returned error can't find the container with id 0ff3994b9bf9dad884e6f6749eefcbef73046610dd42ab7eb19d5c971bd4659f Jul 06 22:47:55.119924 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:55.119769 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d471a2_071d_4716_9402_6bda4d881c6f.slice/crio-2c7e48fdd44570388bd4795dec26400243cfc3762f8e6255098687e5f11ea23b WatchSource:0}: Error finding container 2c7e48fdd44570388bd4795dec26400243cfc3762f8e6255098687e5f11ea23b: Status 404 returned error can't find the container with id 2c7e48fdd44570388bd4795dec26400243cfc3762f8e6255098687e5f11ea23b Jul 06 22:47:55.119924 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.119915 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:47:55.121265 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:55.120822 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c329ad7_e344_4011_85b7_acb16c4fdb13.slice/crio-b332055e34312f123d6a32d495cf32ed35936a1dc1b1be37fd6d2baedfd475fb WatchSource:0}: Error finding container b332055e34312f123d6a32d495cf32ed35936a1dc1b1be37fd6d2baedfd475fb: Status 404 returned error can't find the container with id b332055e34312f123d6a32d495cf32ed35936a1dc1b1be37fd6d2baedfd475fb Jul 06 22:47:55.121326 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.121259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-77556fbd49-v8jdm"] Jul 06 22:47:55.146080 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.146049 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:55.150404 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.150380 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-4hzkn" Jul 06 22:47:55.243324 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.243208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:55.308866 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.308230 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-4hzkn"] Jul 06 22:47:55.312112 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.311759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:55.473418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.473382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:55.573133 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.573032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:55.686018 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.685982 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-4hzkn" event={"ID":"bc362fc8-5494-472c-9b01-122d414ab850","Type":"ContainerStarted","Data":"413d16cef9545f394e0dff3725283e7e77ba30402acab8b6977d081664a84097"} Jul 06 22:47:55.687267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.687240 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7ccm" event={"ID":"4a33cb94-f91f-4a25-85d5-850bd2ab71e3","Type":"ContainerStarted","Data":"9bcaa1faa5cfc980fe58d6c413c0355fe21c13ccdbd0bb68d11f299a00449555"} Jul 06 22:47:55.688575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.688524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jmt2k" event={"ID":"2c329ad7-e344-4011-85b7-acb16c4fdb13","Type":"ContainerStarted","Data":"b332055e34312f123d6a32d495cf32ed35936a1dc1b1be37fd6d2baedfd475fb"} Jul 06 22:47:55.689805 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.689782 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" event={"ID":"69d471a2-071d-4716-9402-6bda4d881c6f","Type":"ContainerStarted","Data":"2c7e48fdd44570388bd4795dec26400243cfc3762f8e6255098687e5f11ea23b"} Jul 06 22:47:55.692026 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.692003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" event={"ID":"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab","Type":"ContainerStarted","Data":"bc7c36fe67e5214073b8586740d472ddf1bcf0d6a034eb8e5285b4dbff64116c"} Jul 06 22:47:55.692137 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.692053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" event={"ID":"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab","Type":"ContainerStarted","Data":"36edd8292bbb268261a2caae4ec612be1349d3c98b7119082026d3ba6620068b"} Jul 06 22:47:55.692137 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.692081 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" event={"ID":"8aa77dd4-6ea9-4e8d-aea9-a385229b30ab","Type":"ContainerStarted","Data":"045d32b4068938f55171321e0ff57660468170f1399e476bfa3f6a23e1e4bbda"} Jul 06 22:47:55.693592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.693522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerStarted","Data":"0ff3994b9bf9dad884e6f6749eefcbef73046610dd42ab7eb19d5c971bd4659f"} Jul 06 22:47:55.694786 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.694768 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cmkgb" event={"ID":"efadaccb-16ab-4bf9-aa7d-93db206c15ab","Type":"ContainerStarted","Data":"8fe88377e6e96e4576ff19d9276d41c35c94bef62bb92439c17f840239837a03"} Jul 06 22:47:55.696947 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.696916 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" event={"ID":"0751e64e-0b6c-4253-b6c3-879f09840d04","Type":"ContainerStarted","Data":"da5777b8c704814fa182838725d56c7dc7ea0fda71407c44a2eaaaab3fc23026"} Jul 06 22:47:55.717912 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.717473 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-gntjg" podStartSLOduration=2.323622279 podStartE2EDuration="3.717402294s" totalImagesPullingTime="1.393780015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:55.715687243 +0000 UTC m=+67.851411819" watchObservedRunningTime="2026-07-06 22:47:55.717402294 +0000 UTC m=+67.853126853" Jul 06 22:47:55.734817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.734758 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-rpszq" podStartSLOduration=2.484412111 podStartE2EDuration="3.734739859s" totalImagesPullingTime="1.250327748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:55.733560397 +0000 UTC m=+67.869284953" watchObservedRunningTime="2026-07-06 22:47:55.734739859 +0000 UTC m=+67.870464413" Jul 06 22:47:55.910562 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:55.910484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.005993 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.005902 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.141617 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.141560 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.258744 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.258669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.345534 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.345465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.510570 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.510454 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7f66d85f8-xlvkl"] Jul 06 22:47:56.540676 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.538756 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f66d85f8-xlvkl"] Jul 06 22:47:56.540676 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.538877 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.545963 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.543466 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 22:47:56.545963 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.543704 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eujn6c29h5m9p\"" Jul 06 22:47:56.545963 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.543890 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 22:47:56.545963 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.544089 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b6zcw\"" Jul 06 22:47:56.545963 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.544303 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 22:47:56.545963 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.544517 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 22:47:56.587236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.586842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27b5e783-00bd-4724-995f-89d634c97d62-audit-log\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.587236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.586897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-client-ca-bundle\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.587236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.586945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-secret-metrics-server-client-certs\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.587236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.586983 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b5e783-00bd-4724-995f-89d634c97d62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.587236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.587015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27b5e783-00bd-4724-995f-89d634c97d62-metrics-server-audit-profiles\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.587236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.587040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-secret-metrics-server-tls\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.587236 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.587081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjgf\" (UniqueName: \"kubernetes.io/projected/27b5e783-00bd-4724-995f-89d634c97d62-kube-api-access-fvjgf\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.689206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.688052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b5e783-00bd-4724-995f-89d634c97d62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.689206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.688107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27b5e783-00bd-4724-995f-89d634c97d62-metrics-server-audit-profiles\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.689206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.688156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-secret-metrics-server-tls\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.689206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.688199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjgf\" (UniqueName: \"kubernetes.io/projected/27b5e783-00bd-4724-995f-89d634c97d62-kube-api-access-fvjgf\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.689206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.688260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27b5e783-00bd-4724-995f-89d634c97d62-audit-log\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.689206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.688303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-client-ca-bundle\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.689206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.688355 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-secret-metrics-server-client-certs\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.692165 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.691103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b5e783-00bd-4724-995f-89d634c97d62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.692165 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.691971 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27b5e783-00bd-4724-995f-89d634c97d62-metrics-server-audit-profiles\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.692165 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.692109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27b5e783-00bd-4724-995f-89d634c97d62-audit-log\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.695321 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.695282 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-client-ca-bundle\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.697157 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.696966 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-secret-metrics-server-client-certs\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.699894 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.699835 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjgf\" (UniqueName: \"kubernetes.io/projected/27b5e783-00bd-4724-995f-89d634c97d62-kube-api-access-fvjgf\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.701184 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.700442 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27b5e783-00bd-4724-995f-89d634c97d62-secret-metrics-server-tls\") pod \"metrics-server-7f66d85f8-xlvkl\" (UID: \"27b5e783-00bd-4724-995f-89d634c97d62\") " pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.708624 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.708169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.708624 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.708449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cmkgb" event={"ID":"efadaccb-16ab-4bf9-aa7d-93db206c15ab","Type":"ContainerStarted","Data":"5ccc31ac3a0764aea04aab9e81c814a749610cffadad38a99df00f6fbb5f2d6a"} Jul 06 22:47:56.708624 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.708502 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cmkgb" event={"ID":"efadaccb-16ab-4bf9-aa7d-93db206c15ab","Type":"ContainerStarted","Data":"21f9ac370c669c4706a55244eaa40fb21e30314f289bb95323d2da9886c0f622"} Jul 06 22:47:56.715249 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.715223 2579 generic.go:356] "Generic (PLEG): container finished" podID="0b90bb90-5d01-4b71-8025-61a1ed6627fd" containerID="39b6940522692241e8f34c412ab1c20974c83117bcf996971c0325320c4b4798" exitCode=0 Jul 06 22:47:56.715460 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.715385 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cnq" event={"ID":"0b90bb90-5d01-4b71-8025-61a1ed6627fd","Type":"ContainerDied","Data":"39b6940522692241e8f34c412ab1c20974c83117bcf996971c0325320c4b4798"} Jul 06 22:47:56.725023 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.724980 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cmkgb" podStartSLOduration=67.631597875 podStartE2EDuration="1m8.724962293s" totalImagesPullingTime="1.093364418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:47:56.723591958 +0000 UTC m=+68.859316515" watchObservedRunningTime="2026-07-06 22:47:56.724962293 +0000 UTC m=+68.860686850" Jul 06 22:47:56.823957 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.823813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.854062 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.853694 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:47:56.932083 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.932035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:56.973547 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.973415 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f"] Jul 06 22:47:56.989016 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.988876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f"] Jul 06 22:47:56.989016 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.988967 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:56.991869 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.991846 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-z62jv\"" Jul 06 22:47:56.992114 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:56.992080 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 22:47:57.014492 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.014399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:57.080835 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.080750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:57.092669 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.092444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/959664d4-bc99-49d9-b3fa-c71f2ea95944-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5qw9f\" (UID: \"959664d4-bc99-49d9-b3fa-c71f2ea95944\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.092669 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.092571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkqj\" (UniqueName: \"kubernetes.io/projected/959664d4-bc99-49d9-b3fa-c71f2ea95944-kube-api-access-mbkqj\") pod \"monitoring-plugin-d5dcff7cf-5qw9f\" (UID: \"959664d4-bc99-49d9-b3fa-c71f2ea95944\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.193412 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.193333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkqj\" (UniqueName: \"kubernetes.io/projected/959664d4-bc99-49d9-b3fa-c71f2ea95944-kube-api-access-mbkqj\") pod \"monitoring-plugin-d5dcff7cf-5qw9f\" (UID: \"959664d4-bc99-49d9-b3fa-c71f2ea95944\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.193566 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.193427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/959664d4-bc99-49d9-b3fa-c71f2ea95944-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5qw9f\" (UID: \"959664d4-bc99-49d9-b3fa-c71f2ea95944\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.193642 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:57.193562 2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 06 22:47:57.193642 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:47:57.193633 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/959664d4-bc99-49d9-b3fa-c71f2ea95944-monitoring-plugin-cert podName:959664d4-bc99-49d9-b3fa-c71f2ea95944 nodeName:}" failed. No retries permitted until 2026-07-06 22:47:57.693611599 +0000 UTC m=+69.829336137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/959664d4-bc99-49d9-b3fa-c71f2ea95944-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-5qw9f" (UID: "959664d4-bc99-49d9-b3fa-c71f2ea95944") : secret "monitoring-plugin-cert" not found Jul 06 22:47:57.206967 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.206935 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkqj\" (UniqueName: \"kubernetes.io/projected/959664d4-bc99-49d9-b3fa-c71f2ea95944-kube-api-access-mbkqj\") pod \"monitoring-plugin-d5dcff7cf-5qw9f\" (UID: \"959664d4-bc99-49d9-b3fa-c71f2ea95944\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.265589 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.265559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:57.516138 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.516098 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7f66d85f8-xlvkl"] Jul 06 22:47:57.697858 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.697823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/959664d4-bc99-49d9-b3fa-c71f2ea95944-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5qw9f\" (UID: \"959664d4-bc99-49d9-b3fa-c71f2ea95944\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.701951 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.701917 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/959664d4-bc99-49d9-b3fa-c71f2ea95944-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-5qw9f\" (UID: \"959664d4-bc99-49d9-b3fa-c71f2ea95944\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.904732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.904688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:47:57.946847 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:57.946814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:58.044500 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.044354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:58.322789 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.322627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:58.421134 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.421054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:58.436183 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:47:58.436146 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b5e783_00bd_4724_995f_89d634c97d62.slice/crio-8883a489341585ccff24631799c660bc4214f26086a78083341609476f549090 WatchSource:0}: Error finding container 8883a489341585ccff24631799c660bc4214f26086a78083341609476f549090: Status 404 returned error can't find the container with id 8883a489341585ccff24631799c660bc4214f26086a78083341609476f549090 Jul 06 22:47:58.497482 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.497414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:58.502045 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.500361 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:47:58.505218 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.505195 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.508937 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.508520 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:47:58.508937 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.508785 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:47:58.510774 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.509915 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:47:58.510774 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510068 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:47:58.510774 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510202 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:47:58.510774 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:47:58.510774 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510412 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:47:58.510774 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510564 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:47:58.510774 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510649 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:47:58.511171 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510944 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:47:58.511171 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.510956 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39ut8t85kt3o4\"" Jul 06 22:47:58.511171 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.511159 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:47:58.512045 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.512029 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:47:58.513388 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.512031 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q5276\"" Jul 06 22:47:58.513388 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.512297 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:47:58.514604 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.514340 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:47:58.519043 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.518751 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607772 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b966l\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-kube-api-access-b966l\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.607968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-config\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.608761 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.608684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710030 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.709990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710099 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-config\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b966l\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-kube-api-access-b966l\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.710621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.710610 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.711876 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.711846 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.715219 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.713579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.715219 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.713795 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.715219 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.714589 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.715532 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.715304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.717594 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.716622 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.717594 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.717015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-web-config\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.717594 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.717298 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.718305 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.718282 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.721703 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.721152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.724968 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.724846 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.725068 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.725034 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.727054 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.726994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:58.727988 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.727948 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.728071 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.727992 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.728071 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.728054 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.728573 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.728550 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.729179 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.728833 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-config-out\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.729179 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.729115 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.731078 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.731032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b966l\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-kube-api-access-b966l\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.731183 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.731089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-config\") pod \"prometheus-k8s-0\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:58.732899 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.732776 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" event={"ID":"27b5e783-00bd-4724-995f-89d634c97d62","Type":"ContainerStarted","Data":"8883a489341585ccff24631799c660bc4214f26086a78083341609476f549090"} Jul 06 22:47:58.823403 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:58.823063 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:47:59.474873 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:59.474829 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:47:59.526783 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:47:59.526733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:00.867594 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:00.867553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:00.967920 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:00.967885 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:01.067831 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:01.067795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:01.168931 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:01.168898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:01.230844 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:01.230757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:01.710728 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:01.710695 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:48:01.724633 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:01.724601 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f"] Jul 06 22:48:02.021813 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:48:02.020100 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d892968_7b68_4874_bd17_8708f4abd501.slice/crio-edaa2034902805332c3e2251b61be79e9d1660d164f5f2c483ede285daf99b05 WatchSource:0}: Error finding container edaa2034902805332c3e2251b61be79e9d1660d164f5f2c483ede285daf99b05: Status 404 returned error can't find the container with id edaa2034902805332c3e2251b61be79e9d1660d164f5f2c483ede285daf99b05 Jul 06 22:48:02.021813 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:48:02.021489 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959664d4_bc99_49d9_b3fa_c71f2ea95944.slice/crio-cd2dfae15d3d10d576c3606031d5702d5dede1b88422204be3c129957343bcef WatchSource:0}: Error finding container cd2dfae15d3d10d576c3606031d5702d5dede1b88422204be3c129957343bcef: Status 404 returned error can't find the container with id cd2dfae15d3d10d576c3606031d5702d5dede1b88422204be3c129957343bcef Jul 06 22:48:02.755470 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.755427 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cnq" event={"ID":"0b90bb90-5d01-4b71-8025-61a1ed6627fd","Type":"ContainerStarted","Data":"e684da7f446e34fa991e999811317da7136878a3273b67eba0a3700c30428b21"} Jul 06 22:48:02.755647 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.755526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n2cnq" event={"ID":"0b90bb90-5d01-4b71-8025-61a1ed6627fd","Type":"ContainerStarted","Data":"793c0bb2fad75fb21dc09ebc12bd511d815cfe0bc50756afe69ddb0a99a65df8"} Jul 06 22:48:02.757856 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.757586 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v7ccm" event={"ID":"4a33cb94-f91f-4a25-85d5-850bd2ab71e3","Type":"ContainerStarted","Data":"19b2c40d423700938c51f3826207b3d1fc6e7ad150e42bb1235efccfed17b76f"} Jul 06 22:48:02.759479 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.759442 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d892968-7b68-4874-bd17-8708f4abd501" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" exitCode=0 Jul 06 22:48:02.759591 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.759531 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e"} Jul 06 22:48:02.759658 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.759592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerStarted","Data":"edaa2034902805332c3e2251b61be79e9d1660d164f5f2c483ede285daf99b05"} Jul 06 22:48:02.761953 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.761927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jmt2k" event={"ID":"2c329ad7-e344-4011-85b7-acb16c4fdb13","Type":"ContainerStarted","Data":"e81a5b5a28e88d8f0a8c8234dc79c3818b012c037636397855204fe7122febbb"} Jul 06 22:48:02.762056 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.761989 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:48:02.765029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.764876 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" event={"ID":"69d471a2-071d-4716-9402-6bda4d881c6f","Type":"ContainerStarted","Data":"2f6754798dd6dda8a24237020425daa7d139f663be9e25c0988752a18a7ade1d"} Jul 06 22:48:02.765029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.764943 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" event={"ID":"69d471a2-071d-4716-9402-6bda4d881c6f","Type":"ContainerStarted","Data":"f72cfa8380bad825b5a87ba5ceefc81071b6a7b7dfb0e964bed8ad2a62698057"} Jul 06 22:48:02.765029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.764984 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" event={"ID":"69d471a2-071d-4716-9402-6bda4d881c6f","Type":"ContainerStarted","Data":"ae5e7f146774cbf4bd1b9149a49a539d697cb1cbb12e55cae50047281b124fe6"} Jul 06 22:48:02.766877 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.766855 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" event={"ID":"27b5e783-00bd-4724-995f-89d634c97d62","Type":"ContainerStarted","Data":"516e5a7fdbeed47cde80f53d91c2eee937eb3557ddacea323202d36cfa21a04f"} Jul 06 22:48:02.768863 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.768714 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerID="336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698" exitCode=0 Jul 06 22:48:02.768863 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.768793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698"} Jul 06 22:48:02.771339 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.771307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" event={"ID":"959664d4-bc99-49d9-b3fa-c71f2ea95944","Type":"ContainerStarted","Data":"cd2dfae15d3d10d576c3606031d5702d5dede1b88422204be3c129957343bcef"} Jul 06 22:48:02.777559 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.777518 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n2cnq" podStartSLOduration=9.351238217 podStartE2EDuration="10.777502265s" totalImagesPullingTime="1.315381298s" totalInitContainerRuntime="110.88275ms" isStatefulPod=true podCreationTimestamp="2026-07-06 22:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:02.776852913 +0000 UTC m=+74.912577490" watchObservedRunningTime="2026-07-06 22:48:02.777502265 +0000 UTC m=+74.913226820" Jul 06 22:48:02.795449 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.795408 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jmt2k" podStartSLOduration=67.896118604 podStartE2EDuration="1m14.795394405s" totalImagesPullingTime="6.899275801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:02.794193486 +0000 UTC m=+74.929918067" watchObservedRunningTime="2026-07-06 22:48:02.795394405 +0000 UTC m=+74.931118961" Jul 06 22:48:02.844320 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:02.844261 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" podStartSLOduration=3.223956925 podStartE2EDuration="6.844230789s" totalImagesPullingTime="3.620273864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:02.842873091 +0000 UTC m=+74.978597648" watchObservedRunningTime="2026-07-06 22:48:02.844230789 +0000 UTC m=+74.979955329" Jul 06 22:48:03.783584 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.783348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" event={"ID":"69d471a2-071d-4716-9402-6bda4d881c6f","Type":"ContainerStarted","Data":"35c61f370c2fbfde98b90872dd2a23178e87fe9ccff989595f0a85a043b1ade3"} Jul 06 22:48:03.783584 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.783433 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" event={"ID":"69d471a2-071d-4716-9402-6bda4d881c6f","Type":"ContainerStarted","Data":"1b0ea4686d7f3920fc3d40ed0ac746560ebac6c36e03a16dd53c2240d22e9791"} Jul 06 22:48:03.783584 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.783475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" event={"ID":"69d471a2-071d-4716-9402-6bda4d881c6f","Type":"ContainerStarted","Data":"db6c6664c39b8d9a5082b2909d0c6508dbcf143e8115e3aa3cac747145ea5616"} Jul 06 22:48:03.784614 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.784545 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:48:03.789732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.788072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" event={"ID":"959664d4-bc99-49d9-b3fa-c71f2ea95944","Type":"ContainerStarted","Data":"dd2bbb28f008670e8994e307207c37aa24e3f256ccf1b450e244befd536180e4"} Jul 06 22:48:03.789732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.789316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:48:03.795836 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.795792 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" Jul 06 22:48:03.812806 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.812739 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" podStartSLOduration=1.775935643 podStartE2EDuration="9.812718522s" totalImagesPullingTime="8.036782879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:03.811809151 +0000 UTC m=+75.947533705" watchObservedRunningTime="2026-07-06 22:48:03.812718522 +0000 UTC m=+75.948443079" Jul 06 22:48:03.813739 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.813697 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v7ccm" podStartSLOduration=67.905674705 podStartE2EDuration="1m14.813683499s" totalImagesPullingTime="6.908008794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:46:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:02.890303327 +0000 UTC m=+75.026027874" watchObservedRunningTime="2026-07-06 22:48:03.813683499 +0000 UTC m=+75.949408055" Jul 06 22:48:03.832171 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:03.832101 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-5qw9f" podStartSLOduration=6.349579902 podStartE2EDuration="7.832070736s" totalImagesPullingTime="1.482490834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:03.830525893 +0000 UTC m=+75.966250465" watchObservedRunningTime="2026-07-06 22:48:03.832070736 +0000 UTC m=+75.967795292" Jul 06 22:48:05.801153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:05.800926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerStarted","Data":"ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28"} Jul 06 22:48:05.801153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:05.801020 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerStarted","Data":"aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc"} Jul 06 22:48:05.801153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:05.801066 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerStarted","Data":"aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0"} Jul 06 22:48:05.801153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:05.801110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerStarted","Data":"41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642"} Jul 06 22:48:05.801752 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:05.801341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerStarted","Data":"eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12"} Jul 06 22:48:05.801752 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:05.801394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerStarted","Data":"6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd"} Jul 06 22:48:05.827525 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:05.827455 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.442471329 podStartE2EDuration="12.82743925s" totalImagesPullingTime="8.37039764s" totalInitContainerRuntime="14.570281ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:05.826101379 +0000 UTC m=+77.961825937" watchObservedRunningTime="2026-07-06 22:48:05.82743925 +0000 UTC m=+77.963163805" Jul 06 22:48:06.811218 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:06.811175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerStarted","Data":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} Jul 06 22:48:06.811525 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:06.811262 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerStarted","Data":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} Jul 06 22:48:07.821939 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:07.821898 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerStarted","Data":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} Jul 06 22:48:07.822420 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:07.822025 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerStarted","Data":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} Jul 06 22:48:07.822420 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:07.822086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerStarted","Data":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} Jul 06 22:48:07.822420 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:07.822156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerStarted","Data":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} Jul 06 22:48:07.880512 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:07.880455 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.981151651 podStartE2EDuration="9.880437613s" totalImagesPullingTime="3.872149938s" totalInitContainerRuntime="27.136024ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:07.87908296 +0000 UTC m=+80.014807555" watchObservedRunningTime="2026-07-06 22:48:07.880437613 +0000 UTC m=+80.016162168" Jul 06 22:48:08.339849 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:08.339783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:08.389170 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:08.389141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:08.532911 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:08.532876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:08.632819 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:08.632718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:08.823921 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:08.823889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:48:09.558351 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:09.558303 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:09.658593 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:09.658543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:10.740462 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:10.740322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:10.808086 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:10.808055 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-77556fbd49-v8jdm" Jul 06 22:48:11.118962 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:11.118888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:11.172630 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:11.172562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:14.737322 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:14.737283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:14.794426 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:14.794394 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:15.868204 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:15.868159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-4hzkn" event={"ID":"bc362fc8-5494-472c-9b01-122d414ab850","Type":"ContainerStarted","Data":"294eef6fd434ec69d92fc18cf5c88561dd34e57928cb02173ee78a937595ad81"} Jul 06 22:48:15.868707 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:15.868503 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-4hzkn" Jul 06 22:48:15.883368 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:15.883340 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-4hzkn" Jul 06 22:48:15.885534 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:15.885484 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-4hzkn" podStartSLOduration=2.213741904 podStartE2EDuration="21.885468278s" totalImagesPullingTime="19.671726374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:15.883966174 +0000 UTC m=+88.019690764" watchObservedRunningTime="2026-07-06 22:48:15.885468278 +0000 UTC m=+88.021192853" Jul 06 22:48:16.228391 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:16.228343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:16.328455 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:16.328422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:16.854587 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:16.854544 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:48:16.854785 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:16.854614 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:48:19.082775 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:19.082731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:19.142500 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:19.142457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:19.196336 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:19.196282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:24.573668 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:24.573633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:24.673701 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:24.673670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:30.693061 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:30.692985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:30.793993 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:30.793948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:33.790980 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:33.790952 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jmt2k" Jul 06 22:48:36.860006 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:36.859975 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:48:36.863948 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:36.863923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7f66d85f8-xlvkl" Jul 06 22:48:39.558958 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:39.558910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:39.612410 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:39.612376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:58.824277 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:58.824242 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:48:58.890381 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:58.890354 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:48:59.075106 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:48:59.075038 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:10.251163 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:10.251098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:11.436972 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:11.436917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:11.639749 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:11.639719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:12.483268 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:12.483233 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:49:12.483680 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:12.483519 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="alertmanager" containerID="cri-o://6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd" gracePeriod=120 Jul 06 22:49:12.483680 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:12.483590 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-metric" containerID="cri-o://aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc" gracePeriod=120 Jul 06 22:49:12.483680 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:12.483602 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-web" containerID="cri-o://41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642" gracePeriod=120 Jul 06 22:49:12.483680 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:12.483625 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="config-reloader" containerID="cri-o://eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12" gracePeriod=120 Jul 06 22:49:12.483680 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:12.483602 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy" containerID="cri-o://aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0" gracePeriod=120 Jul 06 22:49:12.484174 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:12.483684 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="prom-label-proxy" containerID="cri-o://ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28" gracePeriod=120 Jul 06 22:49:13.120158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120110 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerID="ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28" exitCode=0 Jul 06 22:49:13.120158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120150 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerID="aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc" exitCode=0 Jul 06 22:49:13.120158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120157 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerID="aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0" exitCode=0 Jul 06 22:49:13.120158 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28"} Jul 06 22:49:13.120421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120221 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc"} Jul 06 22:49:13.120421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120164 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerID="eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12" exitCode=0 Jul 06 22:49:13.120421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120246 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerID="6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd" exitCode=0 Jul 06 22:49:13.120421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0"} Jul 06 22:49:13.120421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120282 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12"} Jul 06 22:49:13.120421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.120306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd"} Jul 06 22:49:13.721173 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.721151 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:13.876309 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876242 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-web-config\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876309 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876278 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-volume\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876309 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876298 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876553 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876327 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-main-db\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876553 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876453 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-out\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876553 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-tls-assets\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876553 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876528 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99b9d\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-kube-api-access-99b9d\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876756 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876578 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-cluster-tls-config\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876756 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876612 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-metrics-client-ca\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876756 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876642 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:13.876756 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876693 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-web\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.876756 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876735 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-main-tls\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.877009 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.877009 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.876823 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy\") pod \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\" (UID: \"6a48657d-68bf-4f4f-bef0-e187cc0f6c62\") " Jul 06 22:49:13.877112 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.877103 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-main-db\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.877766 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.877736 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:13.877928 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.877907 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:13.879515 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.879487 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-out" (OuterVolumeSpecName: "config-out") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:13.880400 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.880228 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:13.880606 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.880545 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-kube-api-access-99b9d" (OuterVolumeSpecName: "kube-api-access-99b9d") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "kube-api-access-99b9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:13.880606 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.880572 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:13.880836 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.880796 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-volume" (OuterVolumeSpecName: "config-volume") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:13.880967 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.880943 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:13.881039 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.881006 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:13.881179 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.881162 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:13.883633 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.883609 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:13.890503 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.890479 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-web-config" (OuterVolumeSpecName: "web-config") pod "6a48657d-68bf-4f4f-bef0-e187cc0f6c62" (UID: "6a48657d-68bf-4f4f-bef0-e187cc0f6c62"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:13.977857 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977835 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-web-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977857 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977857 2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-volume\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977867 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977876 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-config-out\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977884 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-tls-assets\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977892 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99b9d\" (UniqueName: \"kubernetes.io/projected/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-kube-api-access-99b9d\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977900 2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-cluster-tls-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977908 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-metrics-client-ca\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977916 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977925 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-main-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977935 2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:13.977983 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:13.977943 2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6a48657d-68bf-4f4f-bef0-e187cc0f6c62-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:14.127400 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.127323 2579 generic.go:356] "Generic (PLEG): container finished" podID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerID="41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642" exitCode=0 Jul 06 22:49:14.127524 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.127398 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.127524 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.127408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642"} Jul 06 22:49:14.127524 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.127471 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6a48657d-68bf-4f4f-bef0-e187cc0f6c62","Type":"ContainerDied","Data":"0ff3994b9bf9dad884e6f6749eefcbef73046610dd42ab7eb19d5c971bd4659f"} Jul 06 22:49:14.127524 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.127508 2579 scope.go:117] "RemoveContainer" containerID="ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28" Jul 06 22:49:14.135459 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.135436 2579 scope.go:117] "RemoveContainer" containerID="aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc" Jul 06 22:49:14.142836 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.142821 2579 scope.go:117] "RemoveContainer" containerID="aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0" Jul 06 22:49:14.149200 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.149172 2579 scope.go:117] "RemoveContainer" containerID="41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642" Jul 06 22:49:14.153168 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.153145 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:49:14.157511 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.157486 2579 scope.go:117] "RemoveContainer" containerID="eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12" Jul 06 22:49:14.162812 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.160243 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:49:14.171428 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.171396 2579 scope.go:117] "RemoveContainer" containerID="6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd" Jul 06 22:49:14.178412 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.178394 2579 scope.go:117] "RemoveContainer" containerID="336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698" Jul 06 22:49:14.184855 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.184836 2579 scope.go:117] "RemoveContainer" containerID="ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28" Jul 06 22:49:14.185205 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:14.185100 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28\": container with ID starting with ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28 not found: ID does not exist" containerID="ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28" Jul 06 22:49:14.185205 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185144 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28"} err="failed to get container status \"ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28\": rpc error: code = NotFound desc = could not find container \"ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28\": container with ID starting with ded89893c110bcd6461cea5d09a00bceb01b4a11633bd48bb53f2c291a039b28 not found: ID does not exist" Jul 06 22:49:14.185205 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185179 2579 scope.go:117] "RemoveContainer" containerID="aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc" Jul 06 22:49:14.185425 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:14.185409 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc\": container with ID starting with aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc not found: ID does not exist" containerID="aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc" Jul 06 22:49:14.185465 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185432 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc"} err="failed to get container status \"aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc\": rpc error: code = NotFound desc = could not find container \"aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc\": container with ID starting with aaa501f1dd6edb97172898e6cf9e706c87e9329d37d8cc5085df8406a8cdeadc not found: ID does not exist" Jul 06 22:49:14.185465 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185447 2579 scope.go:117] "RemoveContainer" containerID="aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0" Jul 06 22:49:14.185673 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:14.185659 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0\": container with ID starting with aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0 not found: ID does not exist" containerID="aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0" Jul 06 22:49:14.185707 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185678 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0"} err="failed to get container status \"aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0\": rpc error: code = NotFound desc = could not find container \"aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0\": container with ID starting with aaf550ab4109da1453252ab683e1719856e6119113ecea5b5c1cb1dfb0ecfab0 not found: ID does not exist" Jul 06 22:49:14.185707 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185692 2579 scope.go:117] "RemoveContainer" containerID="41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642" Jul 06 22:49:14.185909 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:14.185893 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642\": container with ID starting with 41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642 not found: ID does not exist" containerID="41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642" Jul 06 22:49:14.185947 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185913 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642"} err="failed to get container status \"41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642\": rpc error: code = NotFound desc = could not find container \"41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642\": container with ID starting with 41eba2f6743264f0e65e9bd3e51b93b40f828bd19caf3cc5d540e121a4aa8642 not found: ID does not exist" Jul 06 22:49:14.185947 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.185926 2579 scope.go:117] "RemoveContainer" containerID="eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12" Jul 06 22:49:14.186158 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:14.186141 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12\": container with ID starting with eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12 not found: ID does not exist" containerID="eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12" Jul 06 22:49:14.186228 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.186166 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12"} err="failed to get container status \"eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12\": rpc error: code = NotFound desc = could not find container \"eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12\": container with ID starting with eec958eb1b14e5fa5b12af7f8791a12cb7e7d560e4e9b8cc1d47711e06af2d12 not found: ID does not exist" Jul 06 22:49:14.186228 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.186185 2579 scope.go:117] "RemoveContainer" containerID="6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd" Jul 06 22:49:14.186415 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:14.186400 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd\": container with ID starting with 6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd not found: ID does not exist" containerID="6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd" Jul 06 22:49:14.186468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.186420 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd"} err="failed to get container status \"6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd\": rpc error: code = NotFound desc = could not find container \"6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd\": container with ID starting with 6851309cdcb3996fdec5731be2c05d35747c36c173a496e3a75328823f4f56cd not found: ID does not exist" Jul 06 22:49:14.186468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.186439 2579 scope.go:117] "RemoveContainer" containerID="336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698" Jul 06 22:49:14.186639 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:14.186623 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698\": container with ID starting with 336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698 not found: ID does not exist" containerID="336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698" Jul 06 22:49:14.186683 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.186643 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698"} err="failed to get container status \"336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698\": rpc error: code = NotFound desc = could not find container \"336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698\": container with ID starting with 336f582aaa22c4e091ad0b6941787130aa26a43d8df64e8cb5fec928a5feb698 not found: ID does not exist" Jul 06 22:49:14.216635 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216611 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:49:14.216915 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216904 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-web" Jul 06 22:49:14.216952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216917 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-web" Jul 06 22:49:14.216952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216931 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="alertmanager" Jul 06 22:49:14.216952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216937 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="alertmanager" Jul 06 22:49:14.216952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216945 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="prom-label-proxy" Jul 06 22:49:14.216952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216951 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="prom-label-proxy" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216961 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="config-reloader" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216966 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="config-reloader" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216972 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216977 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216987 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="init-config-reloader" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.216992 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="init-config-reloader" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217001 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-metric" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217007 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-metric" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217054 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-metric" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217062 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217069 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="prom-label-proxy" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217075 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="config-reloader" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217081 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="alertmanager" Jul 06 22:49:14.217094 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.217090 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" containerName="kube-rbac-proxy-web" Jul 06 22:49:14.221916 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.220153 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.222821 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.222801 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 22:49:14.223191 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223172 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 22:49:14.223315 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223206 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 22:49:14.223315 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223211 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 22:49:14.223315 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223256 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ds5zr\"" Jul 06 22:49:14.223315 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223307 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 22:49:14.223576 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223562 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 22:49:14.223862 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223845 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 22:49:14.223942 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.223910 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 22:49:14.229702 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.229681 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 22:49:14.234655 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.234637 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:49:14.380225 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380225 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed0e763-297f-4d07-ab33-505cb26be55d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380225 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0e763-297f-4d07-ab33-505cb26be55d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380457 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ed0e763-297f-4d07-ab33-505cb26be55d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380457 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ed0e763-297f-4d07-ab33-505cb26be55d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380457 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmrrs\" (UniqueName: \"kubernetes.io/projected/0ed0e763-297f-4d07-ab33-505cb26be55d-kube-api-access-xmrrs\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380457 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380457 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380457 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380657 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380657 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ed0e763-297f-4d07-ab33-505cb26be55d-config-out\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380657 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-web-config\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.380657 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.380519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.427336 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.427311 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a48657d-68bf-4f4f-bef0-e187cc0f6c62" path="/var/lib/kubelet/pods/6a48657d-68bf-4f4f-bef0-e187cc0f6c62/volumes" Jul 06 22:49:14.481206 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481293 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481293 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ed0e763-297f-4d07-ab33-505cb26be55d-config-out\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481387 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-web-config\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481463 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481449 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481496 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed0e763-297f-4d07-ab33-505cb26be55d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481541 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0e763-297f-4d07-ab33-505cb26be55d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481592 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ed0e763-297f-4d07-ab33-505cb26be55d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481641 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ed0e763-297f-4d07-ab33-505cb26be55d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481696 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481641 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmrrs\" (UniqueName: \"kubernetes.io/projected/0ed0e763-297f-4d07-ab33-505cb26be55d-kube-api-access-xmrrs\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481696 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.481811 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.481705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.482472 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.482438 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ed0e763-297f-4d07-ab33-505cb26be55d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484041 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484018 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ed0e763-297f-4d07-ab33-505cb26be55d-config-out\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484588 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484358 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484588 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484365 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-config-volume\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484588 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484363 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484588 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484526 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-web-config\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484831 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484602 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0ed0e763-297f-4d07-ab33-505cb26be55d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484831 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484831 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484641 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.484831 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.484688 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ed0e763-297f-4d07-ab33-505cb26be55d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.485379 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.485359 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ed0e763-297f-4d07-ab33-505cb26be55d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.486173 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.486156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ed0e763-297f-4d07-ab33-505cb26be55d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.492441 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.492424 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmrrs\" (UniqueName: \"kubernetes.io/projected/0ed0e763-297f-4d07-ab33-505cb26be55d-kube-api-access-xmrrs\") pod \"alertmanager-main-0\" (UID: \"0ed0e763-297f-4d07-ab33-505cb26be55d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.531306 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.531275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 22:49:14.676498 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:14.676468 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 22:49:14.679563 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:49:14.679532 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed0e763_297f_4d07_ab33_505cb26be55d.slice/crio-65b97d483d5bf825e93d73dfadbb61a50b938e2f0c189c0cb086a8fae4a5a199 WatchSource:0}: Error finding container 65b97d483d5bf825e93d73dfadbb61a50b938e2f0c189c0cb086a8fae4a5a199: Status 404 returned error can't find the container with id 65b97d483d5bf825e93d73dfadbb61a50b938e2f0c189c0cb086a8fae4a5a199 Jul 06 22:49:15.132871 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:15.132786 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ed0e763-297f-4d07-ab33-505cb26be55d" containerID="3938fd4ebf0d2473ad0c0ed59e7b94ffc87b92a4391dc5c71330d42678cb0f02" exitCode=0 Jul 06 22:49:15.133253 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:15.132874 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerDied","Data":"3938fd4ebf0d2473ad0c0ed59e7b94ffc87b92a4391dc5c71330d42678cb0f02"} Jul 06 22:49:15.133253 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:15.132949 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerStarted","Data":"65b97d483d5bf825e93d73dfadbb61a50b938e2f0c189c0cb086a8fae4a5a199"} Jul 06 22:49:16.141523 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.141490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerStarted","Data":"2dfd5142d95cef0a782fbede4d5f87242d360cc0e55b847cf0757f62d7e7f771"} Jul 06 22:49:16.141948 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.141556 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerStarted","Data":"347c1386f93b8446d73a97eb6dc53426577c982b222b713c3bba4fc3c4b0dbc5"} Jul 06 22:49:16.141948 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.141582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerStarted","Data":"4f51f5962aa53e0f9bbf03b71789360a19122090dca98559647d41c0a25b192a"} Jul 06 22:49:16.141948 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.141607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerStarted","Data":"1cf5b0ca70030cb512a72567993e45832d27445976804cf8c4932c741682e9ca"} Jul 06 22:49:16.141948 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.141636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerStarted","Data":"925500db618c00de855eb3635fecefbeb8d9d079487ddcdf1e89b89259593908"} Jul 06 22:49:16.141948 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.141663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0ed0e763-297f-4d07-ab33-505cb26be55d","Type":"ContainerStarted","Data":"fa2563a6c8a70b57aa1e98059cddac494073cb83c992080dd5ac8651467f08b1"} Jul 06 22:49:16.170409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.170354 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.151772536 podStartE2EDuration="2.170338492s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.565956ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:49:16.167644585 +0000 UTC m=+148.303369162" watchObservedRunningTime="2026-07-06 22:49:16.170338492 +0000 UTC m=+148.306063047" Jul 06 22:49:16.511093 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.511055 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-786bd75985-z9hrf"] Jul 06 22:49:16.513816 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.513794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.516171 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.516148 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 22:49:16.516313 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.516275 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 22:49:16.516423 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.516401 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 22:49:16.516538 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.516446 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vzb54\"" Jul 06 22:49:16.516538 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.516511 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 22:49:16.516646 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.516582 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 22:49:16.522171 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.522149 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 22:49:16.526510 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.526489 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-786bd75985-z9hrf"] Jul 06 22:49:16.604409 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-metrics-client-ca\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.604574 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-telemeter-client-tls\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.604574 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-federate-client-tls\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.604574 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4p7w\" (UniqueName: \"kubernetes.io/projected/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-kube-api-access-k4p7w\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.604732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.604732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604635 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.604732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-secret-telemeter-client\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.604732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.604685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-serving-certs-ca-bundle\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.705883 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.705838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-federate-client-tls\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.706532 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.706485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4p7w\" (UniqueName: \"kubernetes.io/projected/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-kube-api-access-k4p7w\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.706649 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.706551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.707028 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.707003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.707090 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.707060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-secret-telemeter-client\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.707147 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.707091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-serving-certs-ca-bundle\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.707193 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.707179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-metrics-client-ca\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.707268 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.707252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-telemeter-client-tls\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.707536 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.707511 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-telemeter-trusted-ca-bundle\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.708030 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.707998 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-serving-certs-ca-bundle\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.708303 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.708286 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-metrics-client-ca\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.709394 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.709365 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-federate-client-tls\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.709744 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.709718 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.709832 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.709778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-telemeter-client-tls\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.709937 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.709918 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-secret-telemeter-client\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.714505 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.714488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4p7w\" (UniqueName: \"kubernetes.io/projected/cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68-kube-api-access-k4p7w\") pod \"telemeter-client-786bd75985-z9hrf\" (UID: \"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68\") " pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.803656 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.803574 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:49:16.803923 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.803867 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="prometheus" containerID="cri-o://19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" gracePeriod=600 Jul 06 22:49:16.804003 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.803912 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy" containerID="cri-o://c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" gracePeriod=600 Jul 06 22:49:16.804003 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.803932 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="config-reloader" containerID="cri-o://716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" gracePeriod=600 Jul 06 22:49:16.804112 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.803966 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-web" containerID="cri-o://bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" gracePeriod=600 Jul 06 22:49:16.804112 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.803940 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" gracePeriod=600 Jul 06 22:49:16.804249 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.804160 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="thanos-sidecar" containerID="cri-o://d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" gracePeriod=600 Jul 06 22:49:16.825277 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.825254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" Jul 06 22:49:16.975801 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:16.975773 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-786bd75985-z9hrf"] Jul 06 22:49:16.977963 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:49:16.977932 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfa7aeb_39d6_4c9c_aee2_6311fffc2f68.slice/crio-fa6cc71106b15d32f8931f55cc6645a6ff066be5aee907f83dd3f7e14c6b12c8 WatchSource:0}: Error finding container fa6cc71106b15d32f8931f55cc6645a6ff066be5aee907f83dd3f7e14c6b12c8: Status 404 returned error can't find the container with id fa6cc71106b15d32f8931f55cc6645a6ff066be5aee907f83dd3f7e14c6b12c8 Jul 06 22:49:17.058945 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.058887 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.112446 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112333 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-web-config\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.112446 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-2\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.112446 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112433 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-0\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.112716 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112460 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-metrics-client-ca\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.112831 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112806 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:17.112887 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112857 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-1\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.112939 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112901 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-kube-rbac-proxy\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.112939 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-tls-assets\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113037 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.112961 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-serving-certs-ca-bundle\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113037 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113003 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-config\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-db\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113070 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-metrics-client-certs\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113153 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113114 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-thanos-prometheus-http-client-file\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113304 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113167 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-config-out\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113304 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113190 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-grpc-tls\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113304 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113238 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113304 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113264 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-tls\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113304 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113294 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b966l\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-kube-api-access-b966l\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113522 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113320 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-kubelet-serving-ca-bundle\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113522 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113351 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-trusted-ca-bundle\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.113522 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113413 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3d892968-7b68-4874-bd17-8708f4abd501\" (UID: \"3d892968-7b68-4874-bd17-8708f4abd501\") " Jul 06 22:49:17.115497 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.113682 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-metrics-client-ca\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.115497 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.115106 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:17.115497 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.115177 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:17.116534 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.116505 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:17.117182 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.117020 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:17.117549 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.117525 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:17.117735 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.117712 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:17.117857 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.117828 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:17.117923 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.117839 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.118047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.118030 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-kube-api-access-b966l" (OuterVolumeSpecName: "kube-api-access-b966l") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "kube-api-access-b966l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:17.118614 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.118436 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.118614 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.118590 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.119233 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.119210 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-config" (OuterVolumeSpecName: "config") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.119530 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.119478 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.119635 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.119563 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.119778 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.119747 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.122234 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.122084 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:17.122234 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.122171 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.122234 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.122211 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-config-out" (OuterVolumeSpecName: "config-out") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:17.134433 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.134398 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-web-config" (OuterVolumeSpecName: "web-config") pod "3d892968-7b68-4874-bd17-8708f4abd501" (UID: "3d892968-7b68-4874-bd17-8708f4abd501"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:49:17.149706 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149672 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d892968-7b68-4874-bd17-8708f4abd501" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" exitCode=0 Jul 06 22:49:17.149706 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149695 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d892968-7b68-4874-bd17-8708f4abd501" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" exitCode=0 Jul 06 22:49:17.149706 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149702 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d892968-7b68-4874-bd17-8708f4abd501" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" exitCode=0 Jul 06 22:49:17.149706 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149709 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d892968-7b68-4874-bd17-8708f4abd501" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" exitCode=0 Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149714 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d892968-7b68-4874-bd17-8708f4abd501" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" exitCode=0 Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149719 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d892968-7b68-4874-bd17-8708f4abd501" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" exitCode=0 Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149750 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149747 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149863 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149889 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149910 2579 scope.go:117] "RemoveContainer" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.149967 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} Jul 06 22:49:17.150149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.150004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3d892968-7b68-4874-bd17-8708f4abd501","Type":"ContainerDied","Data":"edaa2034902805332c3e2251b61be79e9d1660d164f5f2c483ede285daf99b05"} Jul 06 22:49:17.151392 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.151336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" event={"ID":"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68","Type":"ContainerStarted","Data":"fa6cc71106b15d32f8931f55cc6645a6ff066be5aee907f83dd3f7e14c6b12c8"} Jul 06 22:49:17.157920 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.157904 2579 scope.go:117] "RemoveContainer" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.165218 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.165204 2579 scope.go:117] "RemoveContainer" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.172232 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.172210 2579 scope.go:117] "RemoveContainer" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.175437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.175417 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:49:17.179361 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.179339 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:49:17.182315 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.182301 2579 scope.go:117] "RemoveContainer" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.188855 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.188840 2579 scope.go:117] "RemoveContainer" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.195839 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.195810 2579 scope.go:117] "RemoveContainer" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.202509 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.202492 2579 scope.go:117] "RemoveContainer" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.202779 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:17.202758 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": container with ID starting with 4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6 not found: ID does not exist" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.202825 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.202788 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} err="failed to get container status \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": rpc error: code = NotFound desc = could not find container \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": container with ID starting with 4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6 not found: ID does not exist" Jul 06 22:49:17.202825 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.202806 2579 scope.go:117] "RemoveContainer" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.203047 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:17.203030 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": container with ID starting with c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe not found: ID does not exist" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.203111 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203055 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} err="failed to get container status \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": rpc error: code = NotFound desc = could not find container \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": container with ID starting with c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe not found: ID does not exist" Jul 06 22:49:17.203111 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203079 2579 scope.go:117] "RemoveContainer" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.203409 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:17.203392 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": container with ID starting with bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73 not found: ID does not exist" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.203484 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203417 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} err="failed to get container status \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": rpc error: code = NotFound desc = could not find container \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": container with ID starting with bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73 not found: ID does not exist" Jul 06 22:49:17.203484 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203436 2579 scope.go:117] "RemoveContainer" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.203692 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:17.203676 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": container with ID starting with d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868 not found: ID does not exist" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.203752 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203699 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} err="failed to get container status \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": rpc error: code = NotFound desc = could not find container \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": container with ID starting with d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868 not found: ID does not exist" Jul 06 22:49:17.203752 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203721 2579 scope.go:117] "RemoveContainer" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.203954 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:17.203935 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": container with ID starting with 716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a not found: ID does not exist" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.203988 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203959 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} err="failed to get container status \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": rpc error: code = NotFound desc = could not find container \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": container with ID starting with 716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a not found: ID does not exist" Jul 06 22:49:17.203988 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.203974 2579 scope.go:117] "RemoveContainer" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.204197 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:17.204180 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": container with ID starting with 19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098 not found: ID does not exist" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.204264 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204204 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} err="failed to get container status \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": rpc error: code = NotFound desc = could not find container \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": container with ID starting with 19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098 not found: ID does not exist" Jul 06 22:49:17.204264 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204222 2579 scope.go:117] "RemoveContainer" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.204447 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:49:17.204430 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": container with ID starting with 2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e not found: ID does not exist" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.204486 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204452 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e"} err="failed to get container status \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": rpc error: code = NotFound desc = could not find container \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": container with ID starting with 2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e not found: ID does not exist" Jul 06 22:49:17.204486 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204466 2579 scope.go:117] "RemoveContainer" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.204657 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204640 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} err="failed to get container status \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": rpc error: code = NotFound desc = could not find container \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": container with ID starting with 4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6 not found: ID does not exist" Jul 06 22:49:17.204698 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204658 2579 scope.go:117] "RemoveContainer" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.204858 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204834 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} err="failed to get container status \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": rpc error: code = NotFound desc = could not find container \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": container with ID starting with c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe not found: ID does not exist" Jul 06 22:49:17.204858 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.204858 2579 scope.go:117] "RemoveContainer" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.205380 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.205270 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} err="failed to get container status \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": rpc error: code = NotFound desc = could not find container \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": container with ID starting with bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73 not found: ID does not exist" Jul 06 22:49:17.205380 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.205300 2579 scope.go:117] "RemoveContainer" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.205779 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.205750 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} err="failed to get container status \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": rpc error: code = NotFound desc = could not find container \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": container with ID starting with d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868 not found: ID does not exist" Jul 06 22:49:17.205882 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.205781 2579 scope.go:117] "RemoveContainer" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.206091 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206066 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} err="failed to get container status \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": rpc error: code = NotFound desc = could not find container \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": container with ID starting with 716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a not found: ID does not exist" Jul 06 22:49:17.206091 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206090 2579 scope.go:117] "RemoveContainer" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.206363 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206344 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} err="failed to get container status \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": rpc error: code = NotFound desc = could not find container \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": container with ID starting with 19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098 not found: ID does not exist" Jul 06 22:49:17.206441 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206362 2579 scope.go:117] "RemoveContainer" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.206623 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206603 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e"} err="failed to get container status \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": rpc error: code = NotFound desc = could not find container \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": container with ID starting with 2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e not found: ID does not exist" Jul 06 22:49:17.206716 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206625 2579 scope.go:117] "RemoveContainer" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.206857 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206838 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} err="failed to get container status \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": rpc error: code = NotFound desc = could not find container \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": container with ID starting with 4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6 not found: ID does not exist" Jul 06 22:49:17.206912 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.206858 2579 scope.go:117] "RemoveContainer" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.207065 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207046 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} err="failed to get container status \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": rpc error: code = NotFound desc = could not find container \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": container with ID starting with c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe not found: ID does not exist" Jul 06 22:49:17.207151 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207066 2579 scope.go:117] "RemoveContainer" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.207283 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207263 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} err="failed to get container status \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": rpc error: code = NotFound desc = could not find container \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": container with ID starting with bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73 not found: ID does not exist" Jul 06 22:49:17.207339 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207282 2579 scope.go:117] "RemoveContainer" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.207421 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207404 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:49:17.207516 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207498 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} err="failed to get container status \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": rpc error: code = NotFound desc = could not find container \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": container with ID starting with d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868 not found: ID does not exist" Jul 06 22:49:17.207575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207516 2579 scope.go:117] "RemoveContainer" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.207762 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207746 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} err="failed to get container status \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": rpc error: code = NotFound desc = could not find container \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": container with ID starting with 716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a not found: ID does not exist" Jul 06 22:49:17.207833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207763 2579 scope.go:117] "RemoveContainer" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.207833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207771 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="prometheus" Jul 06 22:49:17.207833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207788 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="prometheus" Jul 06 22:49:17.207833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207807 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="config-reloader" Jul 06 22:49:17.207833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207815 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="config-reloader" Jul 06 22:49:17.207833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207831 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="init-config-reloader" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207839 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="init-config-reloader" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207855 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="thanos-sidecar" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207865 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="thanos-sidecar" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207877 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-thanos" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207885 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-thanos" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207895 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-web" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207904 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-web" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207914 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207922 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207976 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} err="failed to get container status \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": rpc error: code = NotFound desc = could not find container \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": container with ID starting with 19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098 not found: ID does not exist" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.207999 2579 scope.go:117] "RemoveContainer" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208005 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-thanos" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208019 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="config-reloader" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208028 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="thanos-sidecar" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208038 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="prometheus" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208047 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy-web" Jul 06 22:49:17.208103 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208059 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d892968-7b68-4874-bd17-8708f4abd501" containerName="kube-rbac-proxy" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208214 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e"} err="failed to get container status \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": rpc error: code = NotFound desc = could not find container \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": container with ID starting with 2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e not found: ID does not exist" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208232 2579 scope.go:117] "RemoveContainer" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208446 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} err="failed to get container status \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": rpc error: code = NotFound desc = could not find container \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": container with ID starting with 4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6 not found: ID does not exist" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208471 2579 scope.go:117] "RemoveContainer" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208675 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} err="failed to get container status \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": rpc error: code = NotFound desc = could not find container \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": container with ID starting with c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe not found: ID does not exist" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208692 2579 scope.go:117] "RemoveContainer" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208897 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} err="failed to get container status \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": rpc error: code = NotFound desc = could not find container \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": container with ID starting with bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73 not found: ID does not exist" Jul 06 22:49:17.208930 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.208935 2579 scope.go:117] "RemoveContainer" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.209452 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.209157 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} err="failed to get container status \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": rpc error: code = NotFound desc = could not find container \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": container with ID starting with d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868 not found: ID does not exist" Jul 06 22:49:17.209452 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.209175 2579 scope.go:117] "RemoveContainer" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.209452 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.209388 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} err="failed to get container status \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": rpc error: code = NotFound desc = could not find container \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": container with ID starting with 716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a not found: ID does not exist" Jul 06 22:49:17.209452 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.209411 2579 scope.go:117] "RemoveContainer" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.210083 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} err="failed to get container status \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": rpc error: code = NotFound desc = could not find container \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": container with ID starting with 19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098 not found: ID does not exist" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.210111 2579 scope.go:117] "RemoveContainer" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.210901 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e"} err="failed to get container status \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": rpc error: code = NotFound desc = could not find container \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": container with ID starting with 2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e not found: ID does not exist" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.210929 2579 scope.go:117] "RemoveContainer" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215637 2579 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215663 2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-config-out\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215677 2579 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-grpc-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215691 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215712 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215726 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b966l\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-kube-api-access-b966l\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215742 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215756 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215775 2579 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215790 2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-web-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215804 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215818 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.216383 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215837 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215851 2579 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-kube-rbac-proxy\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215864 2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3d892968-7b68-4874-bd17-8708f4abd501-tls-assets\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215881 2579 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d892968-7b68-4874-bd17-8708f4abd501-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215894 2579 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-config\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215909 2579 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3d892968-7b68-4874-bd17-8708f4abd501-prometheus-k8s-db\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.215923 2579 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3d892968-7b68-4874-bd17-8708f4abd501-secret-metrics-client-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.216896 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} err="failed to get container status \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": rpc error: code = NotFound desc = could not find container \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": container with ID starting with 4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6 not found: ID does not exist" Jul 06 22:49:17.217252 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.216936 2579 scope.go:117] "RemoveContainer" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.217652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.217255 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} err="failed to get container status \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": rpc error: code = NotFound desc = could not find container \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": container with ID starting with c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe not found: ID does not exist" Jul 06 22:49:17.217652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.217294 2579 scope.go:117] "RemoveContainer" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.217652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.217401 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.217652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.217562 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} err="failed to get container status \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": rpc error: code = NotFound desc = could not find container \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": container with ID starting with bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73 not found: ID does not exist" Jul 06 22:49:17.217652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.217585 2579 scope.go:117] "RemoveContainer" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.218399 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.218349 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} err="failed to get container status \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": rpc error: code = NotFound desc = could not find container \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": container with ID starting with d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868 not found: ID does not exist" Jul 06 22:49:17.218399 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.218382 2579 scope.go:117] "RemoveContainer" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.218936 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.218906 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} err="failed to get container status \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": rpc error: code = NotFound desc = could not find container \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": container with ID starting with 716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a not found: ID does not exist" Jul 06 22:49:17.219029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.218937 2579 scope.go:117] "RemoveContainer" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.219335 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.219310 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} err="failed to get container status \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": rpc error: code = NotFound desc = could not find container \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": container with ID starting with 19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098 not found: ID does not exist" Jul 06 22:49:17.219404 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.219338 2579 scope.go:117] "RemoveContainer" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.219655 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.219630 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e"} err="failed to get container status \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": rpc error: code = NotFound desc = could not find container \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": container with ID starting with 2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e not found: ID does not exist" Jul 06 22:49:17.219729 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.219665 2579 scope.go:117] "RemoveContainer" containerID="4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6" Jul 06 22:49:17.219996 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.219923 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6"} err="failed to get container status \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": rpc error: code = NotFound desc = could not find container \"4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6\": container with ID starting with 4ad167a812bc4646e02aae05d1ba7d58baa187e8ce88901b7034474af7e5bec6 not found: ID does not exist" Jul 06 22:49:17.219996 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.219949 2579 scope.go:117] "RemoveContainer" containerID="c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe" Jul 06 22:49:17.220178 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 22:49:17.220178 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220156 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q5276\"" Jul 06 22:49:17.220481 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220458 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 22:49:17.220566 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220502 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 22:49:17.220733 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220714 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 22:49:17.220888 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220716 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39ut8t85kt3o4\"" Jul 06 22:49:17.220888 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220727 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 22:49:17.221007 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220919 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe"} err="failed to get container status \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": rpc error: code = NotFound desc = could not find container \"c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe\": container with ID starting with c711c6a99c257c3bf2c7fc6d7858149e1c972415fde2a372dd9d76cacb5a30fe not found: ID does not exist" Jul 06 22:49:17.221007 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.220950 2579 scope.go:117] "RemoveContainer" containerID="bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73" Jul 06 22:49:17.221356 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221317 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73"} err="failed to get container status \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": rpc error: code = NotFound desc = could not find container \"bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73\": container with ID starting with bfd0f523adfeda95ab3cddc19e8ac8923452fa117e95ed5ddcb9344e26ab4d73 not found: ID does not exist" Jul 06 22:49:17.221435 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221359 2579 scope.go:117] "RemoveContainer" containerID="d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868" Jul 06 22:49:17.221683 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221651 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868"} err="failed to get container status \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": rpc error: code = NotFound desc = could not find container \"d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868\": container with ID starting with d1bca7f295cd38f3059fae7f569ab0fa4ac498683286613a6b3e5c956a676868 not found: ID does not exist" Jul 06 22:49:17.221731 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221685 2579 scope.go:117] "RemoveContainer" containerID="716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a" Jul 06 22:49:17.221893 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 22:49:17.222082 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221880 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 22:49:17.222082 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221951 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 22:49:17.222082 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221956 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a"} err="failed to get container status \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": rpc error: code = NotFound desc = could not find container \"716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a\": container with ID starting with 716e730ef21ec95ff7325ba63459305216a82ac1ad1f469480b2f50fca6f5b8a not found: ID does not exist" Jul 06 22:49:17.222082 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.221980 2579 scope.go:117] "RemoveContainer" containerID="19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098" Jul 06 22:49:17.222496 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.222211 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 22:49:17.222496 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.222277 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 22:49:17.222496 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.222276 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 22:49:17.222496 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.222413 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 22:49:17.222817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.222513 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098"} err="failed to get container status \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": rpc error: code = NotFound desc = could not find container \"19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098\": container with ID starting with 19c3f29dcdac13c9d5558659be8881f93dab640cd5199cf5c62272fd45cc5098 not found: ID does not exist" Jul 06 22:49:17.222817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.222534 2579 scope.go:117] "RemoveContainer" containerID="2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e" Jul 06 22:49:17.222817 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.222795 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e"} err="failed to get container status \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": rpc error: code = NotFound desc = could not find container \"2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e\": container with ID starting with 2c97182947e406bec6ff4498229fff24a6812b62296c14336523787eb3cc171e not found: ID does not exist" Jul 06 22:49:17.225361 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.225341 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 22:49:17.228987 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.228958 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 22:49:17.231332 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.231315 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:49:17.316554 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316475 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316554 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/898ed5f6-bdba-464e-907d-2e66784752bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316554 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316642 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c29z9\" (UniqueName: \"kubernetes.io/projected/898ed5f6-bdba-464e-907d-2e66784752bc-kube-api-access-c29z9\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316701 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.316776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316773 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316788 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/898ed5f6-bdba-464e-907d-2e66784752bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-config\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316922 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.317047 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.316984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417538 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417711 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/898ed5f6-bdba-464e-907d-2e66784752bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417711 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417711 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417711 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417711 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417711 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.417711 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417701 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c29z9\" (UniqueName: \"kubernetes.io/projected/898ed5f6-bdba-464e-907d-2e66784752bc-kube-api-access-c29z9\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/898ed5f6-bdba-464e-907d-2e66784752bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417963 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.417975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418058 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.418041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-config\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418580 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.418067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418580 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.418112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418580 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.418169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418580 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.418447 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418773 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.418579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.418773 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.418582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.419521 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.419488 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.419732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.419705 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.420660 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.420632 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/898ed5f6-bdba-464e-907d-2e66784752bc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.420924 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.420900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.421182 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.421158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.422468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.422438 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/898ed5f6-bdba-464e-907d-2e66784752bc-config-out\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.422641 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.422620 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/898ed5f6-bdba-464e-907d-2e66784752bc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.423142 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.423100 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.423232 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.423152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-web-config\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.423334 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.423315 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.424778 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.424751 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.424867 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.424789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-config\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.424867 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.424789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.424978 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.424964 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.425288 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.425268 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/898ed5f6-bdba-464e-907d-2e66784752bc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.427271 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.427250 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c29z9\" (UniqueName: \"kubernetes.io/projected/898ed5f6-bdba-464e-907d-2e66784752bc-kube-api-access-c29z9\") pod \"prometheus-k8s-0\" (UID: \"898ed5f6-bdba-464e-907d-2e66784752bc\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.528485 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.528444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:17.685613 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:17.685585 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 22:49:17.688794 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:49:17.688766 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898ed5f6_bdba_464e_907d_2e66784752bc.slice/crio-d22379cad764962bd328dcc1cff1102a131eb6fd4b8dbf0ae4782c0be58b4edb WatchSource:0}: Error finding container d22379cad764962bd328dcc1cff1102a131eb6fd4b8dbf0ae4782c0be58b4edb: Status 404 returned error can't find the container with id d22379cad764962bd328dcc1cff1102a131eb6fd4b8dbf0ae4782c0be58b4edb Jul 06 22:49:18.158425 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:18.158382 2579 generic.go:356] "Generic (PLEG): container finished" podID="898ed5f6-bdba-464e-907d-2e66784752bc" containerID="7d5565bbbf9929d9d7dc7f7489ae0bba16af543bdd5e7fe6ebd50995da847b2a" exitCode=0 Jul 06 22:49:18.158884 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:18.158473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerDied","Data":"7d5565bbbf9929d9d7dc7f7489ae0bba16af543bdd5e7fe6ebd50995da847b2a"} Jul 06 22:49:18.158884 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:18.158566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerStarted","Data":"d22379cad764962bd328dcc1cff1102a131eb6fd4b8dbf0ae4782c0be58b4edb"} Jul 06 22:49:18.428833 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:18.428803 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d892968-7b68-4874-bd17-8708f4abd501" path="/var/lib/kubelet/pods/3d892968-7b68-4874-bd17-8708f4abd501/volumes" Jul 06 22:49:19.166678 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.166645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerStarted","Data":"6b8e14b19fb2f56c23ff40a6af7abd0164f28cf75085dee4aa6d887d79284feb"} Jul 06 22:49:19.167113 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.166729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerStarted","Data":"6889be9911bbb57cb4939ed48058f8a9253b4d4f01e3621d13de4a82fabd5cdb"} Jul 06 22:49:19.167113 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.166774 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerStarted","Data":"41e12234f4feb6ab9e19667712a8b51e665fe1f95acd3343f1686e4a056eb5dd"} Jul 06 22:49:19.167113 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.166818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerStarted","Data":"5f6f4a0174cbdbce57f1886d971096b6782c2cb6eecc5310fa2dec5894682d99"} Jul 06 22:49:19.167113 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.166861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerStarted","Data":"499c94221d06d75a47a5cb574d788d38e3f8779c77a285d09158a2a5cb1077ed"} Jul 06 22:49:19.167113 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.166896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"898ed5f6-bdba-464e-907d-2e66784752bc","Type":"ContainerStarted","Data":"255332439fd0d6f97c5f6d695ddf9cfad91b04591084d32367ca3eb1ca211abb"} Jul 06 22:49:19.168732 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.168709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" event={"ID":"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68","Type":"ContainerStarted","Data":"faa60dff5b1c12956b6af2c1926ee255dc6fa2fb5dca0377ea8bf3eb9cc691e7"} Jul 06 22:49:19.168832 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.168754 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" event={"ID":"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68","Type":"ContainerStarted","Data":"bc4cfd4e36f55d20cd5a62f55ad529d892e97e258e9ce94645db96746b15baaf"} Jul 06 22:49:19.168832 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.168773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" event={"ID":"cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68","Type":"ContainerStarted","Data":"c3315963c7433bf971b6c44b428e1d16b7c4247c8e220465213951763c7aeeeb"} Jul 06 22:49:19.194618 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.194574 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.17082431 podStartE2EDuration="2.194560991s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.736681ms" isStatefulPod=false podCreationTimestamp="2026-07-06 22:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:49:19.192227179 +0000 UTC m=+151.327951770" watchObservedRunningTime="2026-07-06 22:49:19.194560991 +0000 UTC m=+151.330285545" Jul 06 22:49:19.213064 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:19.213014 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-786bd75985-z9hrf" podStartSLOduration=1.8185293919999999 podStartE2EDuration="3.21299975s" totalImagesPullingTime="1.394470358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:49:19.211070689 +0000 UTC m=+151.346795291" watchObservedRunningTime="2026-07-06 22:49:19.21299975 +0000 UTC m=+151.348724304" Jul 06 22:49:20.003531 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:20.003499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:20.153282 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:20.153246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:20.749329 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:20.749293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:22.528941 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:22.528901 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:49:31.346016 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:31.345983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:31.449440 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:31.449408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:31.508520 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:31.508493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:48.707023 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:48.706981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:48.807144 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:49:48.807100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:17.528678 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:17.528640 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:50:17.544689 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:17.544641 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:50:18.420779 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:18.420751 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 22:50:26.647267 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:26.647238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:26.699860 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:26.699835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:38.631784 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:38.631737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:38.683002 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:38.682969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:42.479246 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:42.479203 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:42.535695 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:42.535669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:53.633297 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:53.633261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:53.684135 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:53.684100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:53.737588 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:53.737553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.568517 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:54.568480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.667287 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:50:54.667249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:03.858437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:03.858396 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:03.909362 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:03.909331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:12.212072 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:12.211982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:24.607855 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:24.607805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:24.709989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:24.709961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:46.281073 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:46.281038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:47.548670 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:47.548635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:48.259776 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:48.259747 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:51:48.265418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:48.265395 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 22:51:48.268692 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:48.268674 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:51:49.844414 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:49.844371 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:49.973976 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:49.973937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:51.503913 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.503878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:51.604464 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.604429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:51.725021 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.724991 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fsw4"] Jul 06 22:51:51.728311 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.728295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:51.730656 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.730636 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 22:51:51.731430 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.731414 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s88w6\"" Jul 06 22:51:51.731487 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.731415 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 22:51:51.735867 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.735846 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fsw4"] Jul 06 22:51:51.843233 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.843170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7966c84f-87d1-4d7c-8ad6-96d6e681df32-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fsw4\" (UID: \"7966c84f-87d1-4d7c-8ad6-96d6e681df32\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:51.843233 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.843213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqxt\" (UniqueName: \"kubernetes.io/projected/7966c84f-87d1-4d7c-8ad6-96d6e681df32-kube-api-access-zvqxt\") pod \"cert-manager-webhook-6486ffd997-9fsw4\" (UID: \"7966c84f-87d1-4d7c-8ad6-96d6e681df32\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:51.943938 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.943913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqxt\" (UniqueName: \"kubernetes.io/projected/7966c84f-87d1-4d7c-8ad6-96d6e681df32-kube-api-access-zvqxt\") pod \"cert-manager-webhook-6486ffd997-9fsw4\" (UID: \"7966c84f-87d1-4d7c-8ad6-96d6e681df32\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:51.944063 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.943977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7966c84f-87d1-4d7c-8ad6-96d6e681df32-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fsw4\" (UID: \"7966c84f-87d1-4d7c-8ad6-96d6e681df32\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:51.954611 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.954589 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7966c84f-87d1-4d7c-8ad6-96d6e681df32-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-9fsw4\" (UID: \"7966c84f-87d1-4d7c-8ad6-96d6e681df32\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:51.954984 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:51.954968 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqxt\" (UniqueName: \"kubernetes.io/projected/7966c84f-87d1-4d7c-8ad6-96d6e681df32-kube-api-access-zvqxt\") pod \"cert-manager-webhook-6486ffd997-9fsw4\" (UID: \"7966c84f-87d1-4d7c-8ad6-96d6e681df32\") " pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:52.050028 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:52.050002 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:52.188554 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:52.188513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-9fsw4"] Jul 06 22:51:52.191020 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:51:52.190997 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7966c84f_87d1_4d7c_8ad6_96d6e681df32.slice/crio-aca38d93cb2549abdaa91a631879d4387137505ac7c64af14c6f805ce379fb0f WatchSource:0}: Error finding container aca38d93cb2549abdaa91a631879d4387137505ac7c64af14c6f805ce379fb0f: Status 404 returned error can't find the container with id aca38d93cb2549abdaa91a631879d4387137505ac7c64af14c6f805ce379fb0f Jul 06 22:51:52.192938 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:52.192921 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:51:52.196795 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:52.196764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:52.304829 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:52.304799 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:52.775845 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:52.775812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" event={"ID":"7966c84f-87d1-4d7c-8ad6-96d6e681df32","Type":"ContainerStarted","Data":"aca38d93cb2549abdaa91a631879d4387137505ac7c64af14c6f805ce379fb0f"} Jul 06 22:51:53.582894 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:53.582837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:53.919919 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:53.919887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:55.790214 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:55.790111 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" event={"ID":"7966c84f-87d1-4d7c-8ad6-96d6e681df32","Type":"ContainerStarted","Data":"0b00adf2582e24ee5def032379550219dec43895c401bdd53b9bbc19eae7c83c"} Jul 06 22:51:55.790214 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:55.790178 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:51:55.813164 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:55.813103 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" podStartSLOduration=1.5294468669999999 podStartE2EDuration="4.813089374s" totalImagesPullingTime="3.283642507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:51:55.811004359 +0000 UTC m=+307.946728910" watchObservedRunningTime="2026-07-06 22:51:55.813089374 +0000 UTC m=+307.948813928" Jul 06 22:51:56.771672 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:56.771640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:56.870989 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:51:56.870964 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:01.796923 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:01.796897 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-9fsw4" Jul 06 22:52:02.326438 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:02.326402 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:02.653508 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:02.653475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:02.753790 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:02.753763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:06.607014 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:06.606984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:06.698458 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:06.698426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:08.280317 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.280283 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s"] Jul 06 22:52:08.283664 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.283648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.286170 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.286152 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 22:52:08.286954 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.286941 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cbcdr\"" Jul 06 22:52:08.287012 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.286945 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:52:08.293985 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.293964 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s"] Jul 06 22:52:08.295429 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.295403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trqfm\" (UniqueName: \"kubernetes.io/projected/ff051f1a-244d-4eba-a642-c31b8249119d-kube-api-access-trqfm\") pod \"openshift-lws-operator-fd8ccff4c-tt92s\" (UID: \"ff051f1a-244d-4eba-a642-c31b8249119d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.295545 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.295493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff051f1a-244d-4eba-a642-c31b8249119d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tt92s\" (UID: \"ff051f1a-244d-4eba-a642-c31b8249119d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.395920 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.395887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff051f1a-244d-4eba-a642-c31b8249119d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tt92s\" (UID: \"ff051f1a-244d-4eba-a642-c31b8249119d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.396086 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.395984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trqfm\" (UniqueName: \"kubernetes.io/projected/ff051f1a-244d-4eba-a642-c31b8249119d-kube-api-access-trqfm\") pod \"openshift-lws-operator-fd8ccff4c-tt92s\" (UID: \"ff051f1a-244d-4eba-a642-c31b8249119d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.396328 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.396289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff051f1a-244d-4eba-a642-c31b8249119d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tt92s\" (UID: \"ff051f1a-244d-4eba-a642-c31b8249119d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.404632 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.404612 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trqfm\" (UniqueName: \"kubernetes.io/projected/ff051f1a-244d-4eba-a642-c31b8249119d-kube-api-access-trqfm\") pod \"openshift-lws-operator-fd8ccff4c-tt92s\" (UID: \"ff051f1a-244d-4eba-a642-c31b8249119d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.592728 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.592646 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" Jul 06 22:52:08.722695 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.722668 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s"] Jul 06 22:52:08.725042 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:52:08.725013 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff051f1a_244d_4eba_a642_c31b8249119d.slice/crio-d90e4f89a6207a9b5e2d4845cea95cdf4bf283e5bf2a22be6f456ce15f2ff511 WatchSource:0}: Error finding container d90e4f89a6207a9b5e2d4845cea95cdf4bf283e5bf2a22be6f456ce15f2ff511: Status 404 returned error can't find the container with id d90e4f89a6207a9b5e2d4845cea95cdf4bf283e5bf2a22be6f456ce15f2ff511 Jul 06 22:52:08.843551 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:08.843483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" event={"ID":"ff051f1a-244d-4eba-a642-c31b8249119d","Type":"ContainerStarted","Data":"d90e4f89a6207a9b5e2d4845cea95cdf4bf283e5bf2a22be6f456ce15f2ff511"} Jul 06 22:52:18.885519 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:18.885483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" event={"ID":"ff051f1a-244d-4eba-a642-c31b8249119d","Type":"ContainerStarted","Data":"cae8df83dd12005d3709aae69eddfea79a04e3e45f163b22f72fe1473ac6ae66"} Jul 06 22:52:18.903276 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:18.903220 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tt92s" podStartSLOduration=1.742459977 podStartE2EDuration="10.903205882s" totalImagesPullingTime="9.160745905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:18.90162673 +0000 UTC m=+331.037351306" watchObservedRunningTime="2026-07-06 22:52:18.903205882 +0000 UTC m=+331.038930436" Jul 06 22:52:19.588762 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:19.588732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:19.686889 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:19.686856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:21.471953 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:21.471917 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:27.688603 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.688570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:27.729113 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.729078 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7"] Jul 06 22:52:27.732689 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.732669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.736010 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.735988 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 22:52:27.736150 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.736040 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 22:52:27.736150 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.736058 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9r88k\"" Jul 06 22:52:27.736150 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.736069 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 22:52:27.740748 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.740725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7"] Jul 06 22:52:27.764006 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.763984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:27.850404 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.850339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvw8d\" (UniqueName: \"kubernetes.io/projected/d8f97de7-d850-40fd-9163-998e29347da7-kube-api-access-wvw8d\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.850404 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.850380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d8f97de7-d850-40fd-9163-998e29347da7-manager-config\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.850404 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.850403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8f97de7-d850-40fd-9163-998e29347da7-metrics-cert\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.850591 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.850543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8f97de7-d850-40fd-9163-998e29347da7-cert\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.951114 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.951082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d8f97de7-d850-40fd-9163-998e29347da7-manager-config\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.951313 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.951133 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8f97de7-d850-40fd-9163-998e29347da7-metrics-cert\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.951313 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.951236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8f97de7-d850-40fd-9163-998e29347da7-cert\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.951313 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.951268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvw8d\" (UniqueName: \"kubernetes.io/projected/d8f97de7-d850-40fd-9163-998e29347da7-kube-api-access-wvw8d\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.951891 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.951860 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d8f97de7-d850-40fd-9163-998e29347da7-manager-config\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.953819 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.953794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d8f97de7-d850-40fd-9163-998e29347da7-cert\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.953946 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.953928 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d8f97de7-d850-40fd-9163-998e29347da7-metrics-cert\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:27.959202 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:27.959176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvw8d\" (UniqueName: \"kubernetes.io/projected/d8f97de7-d850-40fd-9163-998e29347da7-kube-api-access-wvw8d\") pod \"lws-controller-manager-5747d5c666-wkdl7\" (UID: \"d8f97de7-d850-40fd-9163-998e29347da7\") " pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:28.042373 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:28.042336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:28.180927 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:28.180897 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7"] Jul 06 22:52:28.183262 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:52:28.183233 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f97de7_d850_40fd_9163_998e29347da7.slice/crio-072a287b4d12a6bdcf860f6d45b047573acf41c1af5f5b2baa8d9ab2e7048624 WatchSource:0}: Error finding container 072a287b4d12a6bdcf860f6d45b047573acf41c1af5f5b2baa8d9ab2e7048624: Status 404 returned error can't find the container with id 072a287b4d12a6bdcf860f6d45b047573acf41c1af5f5b2baa8d9ab2e7048624 Jul 06 22:52:28.926755 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:28.926723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" event={"ID":"d8f97de7-d850-40fd-9163-998e29347da7","Type":"ContainerStarted","Data":"072a287b4d12a6bdcf860f6d45b047573acf41c1af5f5b2baa8d9ab2e7048624"} Jul 06 22:52:29.932408 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:29.932372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" event={"ID":"d8f97de7-d850-40fd-9163-998e29347da7","Type":"ContainerStarted","Data":"819c0648cdda679df03b9e162d9da3cff987bce94021d08739301a41da74e68c"} Jul 06 22:52:29.932813 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:29.932498 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:29.949679 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:29.949634 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" podStartSLOduration=1.309826035 podStartE2EDuration="2.949621766s" totalImagesPullingTime="1.639795731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:29.947953812 +0000 UTC m=+342.083678366" watchObservedRunningTime="2026-07-06 22:52:29.949621766 +0000 UTC m=+342.085346320" Jul 06 22:52:34.999890 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:34.999854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:35.102794 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:35.102759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:35.876661 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:35.876612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.212787 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.212758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.324317 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.324279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.344561 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.344529 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb"] Jul 06 22:52:36.347155 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.347116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.351268 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.351239 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 22:52:36.351268 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.351252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 22:52:36.351449 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.351255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 22:52:36.351449 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.351379 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 22:52:36.351449 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.351412 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dx78r\"" Jul 06 22:52:36.376080 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.376044 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb"] Jul 06 22:52:36.428235 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.428204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9b14-c631-4931-8083-95b5b40b24d9-manifests\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.428398 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.428238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk2ln\" (UniqueName: \"kubernetes.io/projected/5e2c9b14-c631-4931-8083-95b5b40b24d9-kube-api-access-fk2ln\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.428398 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.428288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e2c9b14-c631-4931-8083-95b5b40b24d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.428398 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.428318 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e2c9b14-c631-4931-8083-95b5b40b24d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.428398 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.428371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9b14-c631-4931-8083-95b5b40b24d9-tmp\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.494189 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.494084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.528947 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.528916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk2ln\" (UniqueName: \"kubernetes.io/projected/5e2c9b14-c631-4931-8083-95b5b40b24d9-kube-api-access-fk2ln\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.529089 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.528959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e2c9b14-c631-4931-8083-95b5b40b24d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.529089 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.528976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e2c9b14-c631-4931-8083-95b5b40b24d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.529089 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.529008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9b14-c631-4931-8083-95b5b40b24d9-tmp\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.529089 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.529056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9b14-c631-4931-8083-95b5b40b24d9-manifests\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.529482 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.529454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9b14-c631-4931-8083-95b5b40b24d9-manifests\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.531469 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.531443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e2c9b14-c631-4931-8083-95b5b40b24d9-tmp\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.531587 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.531565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e2c9b14-c631-4931-8083-95b5b40b24d9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.531663 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.531574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e2c9b14-c631-4931-8083-95b5b40b24d9-webhook-cert\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.540787 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.540761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk2ln\" (UniqueName: \"kubernetes.io/projected/5e2c9b14-c631-4931-8083-95b5b40b24d9-kube-api-access-fk2ln\") pod \"opendatahub-operator-controller-manager-84df5575d5-rljrb\" (UID: \"5e2c9b14-c631-4931-8083-95b5b40b24d9\") " pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.594726 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.594690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.658290 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.658256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:36.691396 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.691365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.811095 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.811075 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb"] Jul 06 22:52:36.813531 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:52:36.813495 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2c9b14_c631_4931_8083_95b5b40b24d9.slice/crio-e5d80c299b6cb6aec16d2d94e7cdeab446ff6ec4f4264b7844aad0933001baf5 WatchSource:0}: Error finding container e5d80c299b6cb6aec16d2d94e7cdeab446ff6ec4f4264b7844aad0933001baf5: Status 404 returned error can't find the container with id e5d80c299b6cb6aec16d2d94e7cdeab446ff6ec4f4264b7844aad0933001baf5 Jul 06 22:52:36.822452 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.822429 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.892738 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.892705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:36.963499 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.963460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" event={"ID":"5e2c9b14-c631-4931-8083-95b5b40b24d9","Type":"ContainerStarted","Data":"e5d80c299b6cb6aec16d2d94e7cdeab446ff6ec4f4264b7844aad0933001baf5"} Jul 06 22:52:36.998028 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:36.998002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:40.939367 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:40.939334 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5747d5c666-wkdl7" Jul 06 22:52:40.986319 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:40.986285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" event={"ID":"5e2c9b14-c631-4931-8083-95b5b40b24d9","Type":"ContainerStarted","Data":"bf8926331f142e765312ae5bce13e703a7a6aadeb62004f2e369f8b3bbf593ee"} Jul 06 22:52:41.993238 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:41.993206 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e2c9b14-c631-4931-8083-95b5b40b24d9" containerID="bf8926331f142e765312ae5bce13e703a7a6aadeb62004f2e369f8b3bbf593ee" exitCode=0 Jul 06 22:52:41.993596 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:41.993264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" event={"ID":"5e2c9b14-c631-4931-8083-95b5b40b24d9","Type":"ContainerDied","Data":"bf8926331f142e765312ae5bce13e703a7a6aadeb62004f2e369f8b3bbf593ee"} Jul 06 22:52:42.999167 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:42.999113 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" event={"ID":"5e2c9b14-c631-4931-8083-95b5b40b24d9","Type":"ContainerStarted","Data":"8a7ceed177906b423a461e779a88435eff5b6af2bb8a0b231e8ca87d429ec7fd"} Jul 06 22:52:42.999521 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:42.999269 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:43.033601 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:43.033532 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" podStartSLOduration=2.253831523 podStartE2EDuration="7.0335088s" totalImagesPullingTime="3.619612087s" totalInitContainerRuntime="1.16006519s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:43.032467453 +0000 UTC m=+355.168192031" watchObservedRunningTime="2026-07-06 22:52:43.0335088 +0000 UTC m=+355.169233356" Jul 06 22:52:47.754695 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:47.754655 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:47.857364 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:47.857329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:54.006855 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:54.006818 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84df5575d5-rljrb" Jul 06 22:52:55.840845 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:55.840758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:55.968892 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:55.968854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:56.572648 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.572616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:56.679858 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.679823 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7"] Jul 06 22:52:56.687437 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.687414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.697834 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.697812 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 22:52:56.698082 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.698068 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 22:52:56.698204 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.698186 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 22:52:56.700065 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.700052 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 22:52:56.707160 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.707141 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-snqg8\"" Jul 06 22:52:56.725175 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.725145 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7"] Jul 06 22:52:56.762691 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.762663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:56.819483 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.819452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-tls-certs\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.819483 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.819484 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-tmp\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.819687 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.819522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nbm\" (UniqueName: \"kubernetes.io/projected/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-kube-api-access-r8nbm\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.839710 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.839617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:56.909064 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.909034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:56.920290 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.920263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-tls-certs\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.920416 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.920295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-tmp\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.920416 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.920326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nbm\" (UniqueName: \"kubernetes.io/projected/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-kube-api-access-r8nbm\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.922543 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.922521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-tmp\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.922642 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.922633 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-tls-certs\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.932835 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.932812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nbm\" (UniqueName: \"kubernetes.io/projected/a438d8b7-3d21-4dcb-a59a-d6e2bd024970-kube-api-access-r8nbm\") pod \"kube-auth-proxy-54f95589f8-hg2l7\" (UID: \"a438d8b7-3d21-4dcb-a59a-d6e2bd024970\") " pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:56.997065 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:56.997032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" Jul 06 22:52:57.017911 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:57.017277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:57.137479 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:57.137453 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7"] Jul 06 22:52:57.139268 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:52:57.139239 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda438d8b7_3d21_4dcb_a59a_d6e2bd024970.slice/crio-bb001e5fad6bfffd52f30062b9e60de0bb897b548fa8e5959cfec4bd25b0925c WatchSource:0}: Error finding container bb001e5fad6bfffd52f30062b9e60de0bb897b548fa8e5959cfec4bd25b0925c: Status 404 returned error can't find the container with id bb001e5fad6bfffd52f30062b9e60de0bb897b548fa8e5959cfec4bd25b0925c Jul 06 22:52:57.141787 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:57.141766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:57.572709 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:57.572679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:57.748203 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:57.748159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:58.069230 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:58.069191 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" event={"ID":"a438d8b7-3d21-4dcb-a59a-d6e2bd024970","Type":"ContainerStarted","Data":"bb001e5fad6bfffd52f30062b9e60de0bb897b548fa8e5959cfec4bd25b0925c"} Jul 06 22:52:58.372718 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:58.372631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:59.088060 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:59.088025 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:59.236923 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:52:59.236886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:00.080100 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:00.080061 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" event={"ID":"a438d8b7-3d21-4dcb-a59a-d6e2bd024970","Type":"ContainerStarted","Data":"b564f409ebe91ed13decf10743c781dc4e14dc7411ec43a7ce56ac20c6e57bbf"} Jul 06 22:53:00.097682 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:00.097619 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54f95589f8-hg2l7" podStartSLOduration=1.5965202 podStartE2EDuration="4.097604075s" totalImagesPullingTime="2.501083875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:00.096275748 +0000 UTC m=+372.232000318" watchObservedRunningTime="2026-07-06 22:53:00.097604075 +0000 UTC m=+372.233328673" Jul 06 22:53:00.535816 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:00.535786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:04.408023 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:04.407986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:04.537786 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:04.537749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:05.462793 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:05.462735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:05.771654 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:05.771566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:05.936858 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:05.936824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:06.856183 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:06.856097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:07.019799 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.019753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:07.029343 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.029310 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8"] Jul 06 22:53:07.033438 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.033412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.041743 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.041716 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 22:53:07.042276 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.042251 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vwb4x\"" Jul 06 22:53:07.051790 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.051765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8"] Jul 06 22:53:07.104314 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.104279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551ce188-8901-45df-8094-9af7fcd37a8c-cert\") pod \"kserve-controller-manager-d6f4b65b8-gwnw8\" (UID: \"551ce188-8901-45df-8094-9af7fcd37a8c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.104568 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.104324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mh5m\" (UniqueName: \"kubernetes.io/projected/551ce188-8901-45df-8094-9af7fcd37a8c-kube-api-access-7mh5m\") pod \"kserve-controller-manager-d6f4b65b8-gwnw8\" (UID: \"551ce188-8901-45df-8094-9af7fcd37a8c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.194284 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.194244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:07.205424 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.205394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mh5m\" (UniqueName: \"kubernetes.io/projected/551ce188-8901-45df-8094-9af7fcd37a8c-kube-api-access-7mh5m\") pod \"kserve-controller-manager-d6f4b65b8-gwnw8\" (UID: \"551ce188-8901-45df-8094-9af7fcd37a8c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.205591 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.205483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551ce188-8901-45df-8094-9af7fcd37a8c-cert\") pod \"kserve-controller-manager-d6f4b65b8-gwnw8\" (UID: \"551ce188-8901-45df-8094-9af7fcd37a8c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.207947 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.207921 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/551ce188-8901-45df-8094-9af7fcd37a8c-cert\") pod \"kserve-controller-manager-d6f4b65b8-gwnw8\" (UID: \"551ce188-8901-45df-8094-9af7fcd37a8c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.233299 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.233270 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mh5m\" (UniqueName: \"kubernetes.io/projected/551ce188-8901-45df-8094-9af7fcd37a8c-kube-api-access-7mh5m\") pod \"kserve-controller-manager-d6f4b65b8-gwnw8\" (UID: \"551ce188-8901-45df-8094-9af7fcd37a8c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.349597 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.349556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:07.383450 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.383418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:07.552894 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:07.552870 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8"] Jul 06 22:53:07.555176 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:53:07.555152 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551ce188_8901_45df_8094_9af7fcd37a8c.slice/crio-1a8842fb440114bf941bb3349fdd90b0537997dc0ee6fa7229f4b0ee49021665 WatchSource:0}: Error finding container 1a8842fb440114bf941bb3349fdd90b0537997dc0ee6fa7229f4b0ee49021665: Status 404 returned error can't find the container with id 1a8842fb440114bf941bb3349fdd90b0537997dc0ee6fa7229f4b0ee49021665 Jul 06 22:53:08.124712 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:08.124676 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" event={"ID":"551ce188-8901-45df-8094-9af7fcd37a8c","Type":"ContainerStarted","Data":"1a8842fb440114bf941bb3349fdd90b0537997dc0ee6fa7229f4b0ee49021665"} Jul 06 22:53:08.613853 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:08.613818 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:08.899746 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:08.899711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:11.140874 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:11.140839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" event={"ID":"551ce188-8901-45df-8094-9af7fcd37a8c","Type":"ContainerStarted","Data":"86497cbc525b73a68fb27d5d1a9b1d07b21adc3a437c08a52c6874defd0450df"} Jul 06 22:53:11.141310 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:11.140973 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:11.171907 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:11.171862 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" podStartSLOduration=2.538566281 podStartE2EDuration="5.171848044s" totalImagesPullingTime="2.633281763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:53:11.170940746 +0000 UTC m=+383.306665299" watchObservedRunningTime="2026-07-06 22:53:11.171848044 +0000 UTC m=+383.307572598" Jul 06 22:53:42.151243 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:42.151213 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-gwnw8" Jul 06 22:53:57.153326 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:57.153290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:57.252136 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:57.252091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:57.327494 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:53:57.327446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:05.535461 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:05.535409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:05.635784 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:05.635735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:26.061889 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:26.061855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:28.350481 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:28.350433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:28.452522 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:28.452484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:29.931782 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:29.931733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:31.282581 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:31.282548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:31.687355 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:31.687322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:32.722464 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:32.722425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:32.824579 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:32.824550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:34.315695 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:34.315663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:35.331973 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:35.331942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:36.680439 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:36.680405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:37.948868 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:37.948837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:38.332771 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.332693 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46"] Jul 06 22:54:38.336075 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.336058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" Jul 06 22:54:38.338728 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.338698 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 22:54:38.338868 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.338701 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7xhr7\"" Jul 06 22:54:38.338868 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.338706 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 22:54:38.338868 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.338769 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 22:54:38.347885 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.347863 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46"] Jul 06 22:54:38.451738 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.451709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttbst\" (UniqueName: \"kubernetes.io/projected/8e5ec393-e81e-4156-b9f3-fd75651398ae-kube-api-access-ttbst\") pod \"dns-operator-controller-manager-864dd78c4c-79z46\" (UID: \"8e5ec393-e81e-4156-b9f3-fd75651398ae\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" Jul 06 22:54:38.552683 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.552650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttbst\" (UniqueName: \"kubernetes.io/projected/8e5ec393-e81e-4156-b9f3-fd75651398ae-kube-api-access-ttbst\") pod \"dns-operator-controller-manager-864dd78c4c-79z46\" (UID: \"8e5ec393-e81e-4156-b9f3-fd75651398ae\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" Jul 06 22:54:38.570688 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.570661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttbst\" (UniqueName: \"kubernetes.io/projected/8e5ec393-e81e-4156-b9f3-fd75651398ae-kube-api-access-ttbst\") pod \"dns-operator-controller-manager-864dd78c4c-79z46\" (UID: \"8e5ec393-e81e-4156-b9f3-fd75651398ae\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" Jul 06 22:54:38.646759 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.646728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" Jul 06 22:54:38.785468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.785013 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46"] Jul 06 22:54:38.787604 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:54:38.787569 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5ec393_e81e_4156_b9f3_fd75651398ae.slice/crio-03b64cd77b81c9a6c393ada61c35874e27693ca865b084dbc0e2d7bfb5e3cb20 WatchSource:0}: Error finding container 03b64cd77b81c9a6c393ada61c35874e27693ca865b084dbc0e2d7bfb5e3cb20: Status 404 returned error can't find the container with id 03b64cd77b81c9a6c393ada61c35874e27693ca865b084dbc0e2d7bfb5e3cb20 Jul 06 22:54:38.865591 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.865558 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:38.967669 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:38.967577 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:39.066316 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:39.066288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:39.166251 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:39.166218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:39.557763 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:39.557723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" event={"ID":"8e5ec393-e81e-4156-b9f3-fd75651398ae","Type":"ContainerStarted","Data":"03b64cd77b81c9a6c393ada61c35874e27693ca865b084dbc0e2d7bfb5e3cb20"} Jul 06 22:54:40.561715 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:40.561688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:40.563952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:40.563926 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" event={"ID":"8e5ec393-e81e-4156-b9f3-fd75651398ae","Type":"ContainerStarted","Data":"8017ffe55d63c04639b6696c6a897cdd74d481452ded725af5464c8adc4a8ff4"} Jul 06 22:54:40.564081 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:40.563995 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" Jul 06 22:54:40.582873 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:40.582833 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" podStartSLOduration=0.982518703 podStartE2EDuration="2.582818312s" totalImagesPullingTime="1.600299609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:40.58109822 +0000 UTC m=+472.716822802" watchObservedRunningTime="2026-07-06 22:54:40.582818312 +0000 UTC m=+472.718542867" Jul 06 22:54:40.649569 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:40.649533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:40.722571 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:40.722537 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:40.835381 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:40.835350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:41.002818 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:41.002735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:50.667652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:50.667619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:50.828564 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:50.828531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:51.571788 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:51.571760 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-79z46" Jul 06 22:54:53.552850 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:53.552815 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:53.662438 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:53.662405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:54.853100 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:54.853064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:54.948119 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:54.948086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:55.953555 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:55.953514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:56.054582 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:54:56.054550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:01.720936 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:01.720899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:01.825674 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:01.825640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:07.435146 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:07.435086 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:07.534634 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:07.534603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:11.133773 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:11.133736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:12.101211 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:12.101176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:12.202238 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:12.202186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:12.484952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:12.484913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:12.552968 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:12.552937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:15.912564 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:15.912528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:16.006086 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:16.006035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:17.351845 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:17.351811 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:17.453118 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:17.453079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:31.209829 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:31.209790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:31.300654 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:31.300615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:32.944239 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:32.944202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:33.048673 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.048643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:33.092109 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.092063 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:55:33.094944 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.094917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.097072 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.097048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jmlxb\"" Jul 06 22:55:33.097212 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.097081 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 22:55:33.104926 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.104903 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:55:33.118575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.118546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:33.135855 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.135822 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:55:33.256790 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.256690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzxd\" (UniqueName: \"kubernetes.io/projected/7af9668c-1492-4f56-b2cc-5a839e5f1029-kube-api-access-8pzxd\") pod \"limitador-limitador-54d6d988d5-m4d6x\" (UID: \"7af9668c-1492-4f56-b2cc-5a839e5f1029\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.256790 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.256747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7af9668c-1492-4f56-b2cc-5a839e5f1029-config-file\") pod \"limitador-limitador-54d6d988d5-m4d6x\" (UID: \"7af9668c-1492-4f56-b2cc-5a839e5f1029\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.357627 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.357583 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzxd\" (UniqueName: \"kubernetes.io/projected/7af9668c-1492-4f56-b2cc-5a839e5f1029-kube-api-access-8pzxd\") pod \"limitador-limitador-54d6d988d5-m4d6x\" (UID: \"7af9668c-1492-4f56-b2cc-5a839e5f1029\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.357826 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.357638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7af9668c-1492-4f56-b2cc-5a839e5f1029-config-file\") pod \"limitador-limitador-54d6d988d5-m4d6x\" (UID: \"7af9668c-1492-4f56-b2cc-5a839e5f1029\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.358327 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.358303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7af9668c-1492-4f56-b2cc-5a839e5f1029-config-file\") pod \"limitador-limitador-54d6d988d5-m4d6x\" (UID: \"7af9668c-1492-4f56-b2cc-5a839e5f1029\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.366092 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.366068 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzxd\" (UniqueName: \"kubernetes.io/projected/7af9668c-1492-4f56-b2cc-5a839e5f1029-kube-api-access-8pzxd\") pod \"limitador-limitador-54d6d988d5-m4d6x\" (UID: \"7af9668c-1492-4f56-b2cc-5a839e5f1029\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.406099 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.406060 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:33.539003 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.538974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:55:33.541295 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:55:33.541269 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af9668c_1492_4f56_b2cc_5a839e5f1029.slice/crio-385b6d24388e5e66911a288767f682a44b19749dfe4a7e0baa0ced22581653e1 WatchSource:0}: Error finding container 385b6d24388e5e66911a288767f682a44b19749dfe4a7e0baa0ced22581653e1: Status 404 returned error can't find the container with id 385b6d24388e5e66911a288767f682a44b19749dfe4a7e0baa0ced22581653e1 Jul 06 22:55:33.775718 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.775680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:33.816202 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.816103 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" event={"ID":"7af9668c-1492-4f56-b2cc-5a839e5f1029","Type":"ContainerStarted","Data":"385b6d24388e5e66911a288767f682a44b19749dfe4a7e0baa0ced22581653e1"} Jul 06 22:55:33.832442 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:33.832415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:36.834873 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:36.834838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" event={"ID":"7af9668c-1492-4f56-b2cc-5a839e5f1029","Type":"ContainerStarted","Data":"9e9235089af5b915cf734e7412fd8a76b055504e51c44fdb7aba6dd1a5bcc1be"} Jul 06 22:55:36.835291 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:36.834976 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:36.850641 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:36.850593 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" podStartSLOduration=1.090255951 podStartE2EDuration="3.850577866s" totalImagesPullingTime="2.760321915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:36.848739518 +0000 UTC m=+528.984464108" watchObservedRunningTime="2026-07-06 22:55:36.850577866 +0000 UTC m=+528.986302421" Jul 06 22:55:47.840972 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:47.840896 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-m4d6x" Jul 06 22:55:49.014468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:49.014435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:49.119371 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:49.119340 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:49.455766 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:55:49.455736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:01.589075 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:01.589036 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:01.772675 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:01.772641 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:02.088531 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.088499 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86c899c57c-l7frt"] Jul 06 22:56:02.092075 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.092055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86c899c57c-l7frt" Jul 06 22:56:02.095144 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.095115 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zg7r9\"" Jul 06 22:56:02.100370 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.100348 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86c899c57c-l7frt"] Jul 06 22:56:02.117627 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.117604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4wn\" (UniqueName: \"kubernetes.io/projected/12938bb1-92a7-4cc6-9996-f4b73f9d57cd-kube-api-access-wn4wn\") pod \"authorino-86c899c57c-l7frt\" (UID: \"12938bb1-92a7-4cc6-9996-f4b73f9d57cd\") " pod="kuadrant-system/authorino-86c899c57c-l7frt" Jul 06 22:56:02.218895 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.218864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4wn\" (UniqueName: \"kubernetes.io/projected/12938bb1-92a7-4cc6-9996-f4b73f9d57cd-kube-api-access-wn4wn\") pod \"authorino-86c899c57c-l7frt\" (UID: \"12938bb1-92a7-4cc6-9996-f4b73f9d57cd\") " pod="kuadrant-system/authorino-86c899c57c-l7frt" Jul 06 22:56:02.226835 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.226808 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4wn\" (UniqueName: \"kubernetes.io/projected/12938bb1-92a7-4cc6-9996-f4b73f9d57cd-kube-api-access-wn4wn\") pod \"authorino-86c899c57c-l7frt\" (UID: \"12938bb1-92a7-4cc6-9996-f4b73f9d57cd\") " pod="kuadrant-system/authorino-86c899c57c-l7frt" Jul 06 22:56:02.267486 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.267453 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86c899c57c-l7frt"] Jul 06 22:56:02.267740 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.267728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86c899c57c-l7frt" Jul 06 22:56:02.398514 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.398490 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86c899c57c-l7frt"] Jul 06 22:56:02.400699 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:56:02.400674 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12938bb1_92a7_4cc6_9996_f4b73f9d57cd.slice/crio-bd93337c05af2c42dc57c51b0e4a321a0c875d37810f5a6ff31d635addf01e8e WatchSource:0}: Error finding container bd93337c05af2c42dc57c51b0e4a321a0c875d37810f5a6ff31d635addf01e8e: Status 404 returned error can't find the container with id bd93337c05af2c42dc57c51b0e4a321a0c875d37810f5a6ff31d635addf01e8e Jul 06 22:56:02.964707 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:02.964655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86c899c57c-l7frt" event={"ID":"12938bb1-92a7-4cc6-9996-f4b73f9d57cd","Type":"ContainerStarted","Data":"bd93337c05af2c42dc57c51b0e4a321a0c875d37810f5a6ff31d635addf01e8e"} Jul 06 22:56:05.982481 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:05.982441 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86c899c57c-l7frt" event={"ID":"12938bb1-92a7-4cc6-9996-f4b73f9d57cd","Type":"ContainerStarted","Data":"9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158"} Jul 06 22:56:05.982919 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:05.982553 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-86c899c57c-l7frt" podUID="12938bb1-92a7-4cc6-9996-f4b73f9d57cd" containerName="authorino" containerID="cri-o://9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158" gracePeriod=30 Jul 06 22:56:05.997358 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:05.997316 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-86c899c57c-l7frt" podStartSLOduration=1.1541805059999999 podStartE2EDuration="3.997297371s" totalImagesPullingTime="2.843116865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:56:05.996560538 +0000 UTC m=+558.132285093" watchObservedRunningTime="2026-07-06 22:56:05.997297371 +0000 UTC m=+558.133021926" Jul 06 22:56:06.227089 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.227065 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86c899c57c-l7frt" Jul 06 22:56:06.360845 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.360754 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn4wn\" (UniqueName: \"kubernetes.io/projected/12938bb1-92a7-4cc6-9996-f4b73f9d57cd-kube-api-access-wn4wn\") pod \"12938bb1-92a7-4cc6-9996-f4b73f9d57cd\" (UID: \"12938bb1-92a7-4cc6-9996-f4b73f9d57cd\") " Jul 06 22:56:06.362952 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.362930 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12938bb1-92a7-4cc6-9996-f4b73f9d57cd-kube-api-access-wn4wn" (OuterVolumeSpecName: "kube-api-access-wn4wn") pod "12938bb1-92a7-4cc6-9996-f4b73f9d57cd" (UID: "12938bb1-92a7-4cc6-9996-f4b73f9d57cd"). InnerVolumeSpecName "kube-api-access-wn4wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:56:06.462317 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.462284 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn4wn\" (UniqueName: \"kubernetes.io/projected/12938bb1-92a7-4cc6-9996-f4b73f9d57cd-kube-api-access-wn4wn\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 22:56:06.989069 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.989033 2579 generic.go:356] "Generic (PLEG): container finished" podID="12938bb1-92a7-4cc6-9996-f4b73f9d57cd" containerID="9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158" exitCode=0 Jul 06 22:56:06.989069 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.989070 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86c899c57c-l7frt" Jul 06 22:56:06.989653 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.989110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86c899c57c-l7frt" event={"ID":"12938bb1-92a7-4cc6-9996-f4b73f9d57cd","Type":"ContainerDied","Data":"9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158"} Jul 06 22:56:06.989653 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.989177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86c899c57c-l7frt" event={"ID":"12938bb1-92a7-4cc6-9996-f4b73f9d57cd","Type":"ContainerDied","Data":"bd93337c05af2c42dc57c51b0e4a321a0c875d37810f5a6ff31d635addf01e8e"} Jul 06 22:56:06.989653 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.989202 2579 scope.go:117] "RemoveContainer" containerID="9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158" Jul 06 22:56:06.998261 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.998237 2579 scope.go:117] "RemoveContainer" containerID="9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158" Jul 06 22:56:06.998547 ip-10-0-134-52 kubenswrapper[2579]: E0706 22:56:06.998526 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158\": container with ID starting with 9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158 not found: ID does not exist" containerID="9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158" Jul 06 22:56:06.998621 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:06.998554 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158"} err="failed to get container status \"9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158\": rpc error: code = NotFound desc = could not find container \"9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158\": container with ID starting with 9207c23fd809ba27678065e8c20f683760ab6c7db17dfc3fbb44cff69e768158 not found: ID does not exist" Jul 06 22:56:07.004396 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:07.004368 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86c899c57c-l7frt"] Jul 06 22:56:07.006112 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:07.006087 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-86c899c57c-l7frt"] Jul 06 22:56:08.321362 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:08.321326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:08.403099 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:08.403068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:08.429261 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:08.429233 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12938bb1-92a7-4cc6-9996-f4b73f9d57cd" path="/var/lib/kubelet/pods/12938bb1-92a7-4cc6-9996-f4b73f9d57cd/volumes" Jul 06 22:56:08.461782 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:08.461733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:08.534032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:08.533983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:11.117287 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:11.117252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:11.210917 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:11.210884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.003685 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:43.003642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.413119 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:43.413084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.693630 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:43.693546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.827418 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:43.827385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.927712 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:43.927669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:44.078468 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:44.078388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:56:48.291724 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:48.291692 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:56:48.297843 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:48.297820 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 22:56:53.903934 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:53.903901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:55.125174 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:56:55.125143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:32.001181 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:32.001141 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:32.390890 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:32.390795 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:57:42.981032 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.981000 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4"] Jul 06 22:57:42.981423 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.981409 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12938bb1-92a7-4cc6-9996-f4b73f9d57cd" containerName="authorino" Jul 06 22:57:42.981423 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.981424 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="12938bb1-92a7-4cc6-9996-f4b73f9d57cd" containerName="authorino" Jul 06 22:57:42.981516 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.981484 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="12938bb1-92a7-4cc6-9996-f4b73f9d57cd" containerName="authorino" Jul 06 22:57:42.984847 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.984829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:42.988205 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.988181 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 22:57:42.988318 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.988216 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 22:57:42.988318 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.988181 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 22:57:42.988318 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.988282 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bs2s8\"" Jul 06 22:57:42.995148 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:42.995109 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4"] Jul 06 22:57:43.006809 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.006780 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:43.039051 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.039013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.039051 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.039055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.039271 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.039187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7t2\" (UniqueName: \"kubernetes.io/projected/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-kube-api-access-hd7t2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.039271 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.039222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.039271 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.039254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.039382 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.039301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140066 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140066 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140324 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140324 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140152 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140324 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140172 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140324 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7t2\" (UniqueName: \"kubernetes.io/projected/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-kube-api-access-hd7t2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140596 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140566 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140681 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140604 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.140681 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.140655 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.142510 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.142480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.142728 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.142711 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.153517 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.153493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7t2\" (UniqueName: \"kubernetes.io/projected/53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d-kube-api-access-hd7t2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-prcp4\" (UID: \"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.295828 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.295728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:43.431540 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.431509 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4"] Jul 06 22:57:43.433375 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:57:43.433346 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a5d3dc_3931_47f3_bd71_48ae9dbd4f8d.slice/crio-d8c77334ed7a4984417231dfabad21b74489f9b1f73b64f8fc0d507669afd9b7 WatchSource:0}: Error finding container d8c77334ed7a4984417231dfabad21b74489f9b1f73b64f8fc0d507669afd9b7: Status 404 returned error can't find the container with id d8c77334ed7a4984417231dfabad21b74489f9b1f73b64f8fc0d507669afd9b7 Jul 06 22:57:43.434686 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.434667 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:57:43.453428 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.453395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" event={"ID":"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d","Type":"ContainerStarted","Data":"d8c77334ed7a4984417231dfabad21b74489f9b1f73b64f8fc0d507669afd9b7"} Jul 06 22:57:43.775887 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:43.775849 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:57:45.471623 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:45.471585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" event={"ID":"53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d","Type":"ContainerStarted","Data":"c58e94420a2076223dcb16527f04bab0fb2947da5280834dfdc1c73524dac30a"} Jul 06 22:57:45.489405 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:45.489356 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" podStartSLOduration=2.473479987 podStartE2EDuration="3.489341252s" totalImagesPullingTime="1.015861265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:45.487516918 +0000 UTC m=+657.623241510" watchObservedRunningTime="2026-07-06 22:57:45.489341252 +0000 UTC m=+657.625065807" Jul 06 22:57:50.579762 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:50.579709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:51.780571 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:51.780532 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:57:53.296149 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:53.296098 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:53.296555 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:53.296183 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:53.308603 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:53.308579 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:53.522426 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:53.522398 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-prcp4" Jul 06 22:57:55.664312 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:55.664281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:55.721646 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:57:55.721608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:10.375575 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:10.375536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:11.885773 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:11.885731 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:58:22.379280 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.379241 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2"] Jul 06 22:58:22.384704 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.384679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.387041 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.387024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 22:58:22.393449 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.393422 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2"] Jul 06 22:58:22.400955 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.400919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:22.507796 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.507764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjrjd\" (UniqueName: \"kubernetes.io/projected/6461e12c-ebf2-4d98-b60c-62904ad2eea0-kube-api-access-tjrjd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.507796 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.507806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.508029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.507823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.508029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.507908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.508029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.507952 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6461e12c-ebf2-4d98-b60c-62904ad2eea0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.508029 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.507989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609035 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609035 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609341 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609341 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6461e12c-ebf2-4d98-b60c-62904ad2eea0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609341 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609341 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjrjd\" (UniqueName: \"kubernetes.io/projected/6461e12c-ebf2-4d98-b60c-62904ad2eea0-kube-api-access-tjrjd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609554 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609554 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.609652 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.609582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.611323 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.611304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6461e12c-ebf2-4d98-b60c-62904ad2eea0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.611572 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.611552 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6461e12c-ebf2-4d98-b60c-62904ad2eea0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.617688 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.617661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjrjd\" (UniqueName: \"kubernetes.io/projected/6461e12c-ebf2-4d98-b60c-62904ad2eea0-kube-api-access-tjrjd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5lmf2\" (UID: \"6461e12c-ebf2-4d98-b60c-62904ad2eea0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.696347 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.696315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:22.832379 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:22.832345 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2"] Jul 06 22:58:22.833960 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:58:22.833934 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6461e12c_ebf2_4d98_b60c_62904ad2eea0.slice/crio-f8d8507f6445317760ac1ba8b67856902b52bf67b90f45c293937c0cb37428d0 WatchSource:0}: Error finding container f8d8507f6445317760ac1ba8b67856902b52bf67b90f45c293937c0cb37428d0: Status 404 returned error can't find the container with id f8d8507f6445317760ac1ba8b67856902b52bf67b90f45c293937c0cb37428d0 Jul 06 22:58:23.653700 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:23.653663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" event={"ID":"6461e12c-ebf2-4d98-b60c-62904ad2eea0","Type":"ContainerStarted","Data":"500a4d85c2c771d14817d0ff3b8e167479a3b572cdefd4bf5c03cbeec002f099"} Jul 06 22:58:23.654197 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:23.653722 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" event={"ID":"6461e12c-ebf2-4d98-b60c-62904ad2eea0","Type":"ContainerStarted","Data":"f8d8507f6445317760ac1ba8b67856902b52bf67b90f45c293937c0cb37428d0"} Jul 06 22:58:23.671907 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:23.671854 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" podStartSLOduration=1.5131722079999999 podStartE2EDuration="1.671837234s" totalImagesPullingTime="158.665026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:58:23.670887861 +0000 UTC m=+695.806612417" watchObservedRunningTime="2026-07-06 22:58:23.671837234 +0000 UTC m=+695.807561790" Jul 06 22:58:24.479039 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:24.479005 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:58:32.696870 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:32.696831 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:32.697329 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:32.696894 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:32.709262 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:32.709238 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:33.713678 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:33.713603 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5lmf2" Jul 06 22:58:42.192099 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.192019 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9"] Jul 06 22:58:42.195787 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.195769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.200542 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.200522 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 22:58:42.214545 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.214522 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9"] Jul 06 22:58:42.225443 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.225409 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:42.301514 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.301473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.301705 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.301536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2t64\" (UniqueName: \"kubernetes.io/projected/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-kube-api-access-d2t64\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.301705 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.301596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.301705 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.301638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.301705 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.301671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.301843 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.301733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402395 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402573 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2t64\" (UniqueName: \"kubernetes.io/projected/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-kube-api-access-d2t64\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402573 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402573 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402573 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402573 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402846 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402815 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402846 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402842 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.402995 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.402938 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.404847 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.404825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.404933 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.404909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.409986 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.409967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2t64\" (UniqueName: \"kubernetes.io/projected/a73cee4b-73fa-4d15-90d6-c87c17a94a4f-kube-api-access-d2t64\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9\" (UID: \"a73cee4b-73fa-4d15-90d6-c87c17a94a4f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.505104 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.505009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:42.580274 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.580245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:42.638625 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.638601 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9"] Jul 06 22:58:42.640603 ip-10-0-134-52 kubenswrapper[2579]: W0706 22:58:42.640579 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda73cee4b_73fa_4d15_90d6_c87c17a94a4f.slice/crio-e4a18023a38af9841a1a3be8c7c4b61f5fbabe722eb424622959a405dae3c9b2 WatchSource:0}: Error finding container e4a18023a38af9841a1a3be8c7c4b61f5fbabe722eb424622959a405dae3c9b2: Status 404 returned error can't find the container with id e4a18023a38af9841a1a3be8c7c4b61f5fbabe722eb424622959a405dae3c9b2 Jul 06 22:58:42.745484 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:42.745449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" event={"ID":"a73cee4b-73fa-4d15-90d6-c87c17a94a4f","Type":"ContainerStarted","Data":"e4a18023a38af9841a1a3be8c7c4b61f5fbabe722eb424622959a405dae3c9b2"} Jul 06 22:58:43.752801 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:43.752764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" event={"ID":"a73cee4b-73fa-4d15-90d6-c87c17a94a4f","Type":"ContainerStarted","Data":"b998836cd4a979adadb9132dde6e6ae0bd088826e38bd63f4c3e6c8377b02d28"} Jul 06 22:58:43.770068 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:43.770017 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" podStartSLOduration=1.5877857770000001 podStartE2EDuration="1.770000392s" totalImagesPullingTime="182.214615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:58:43.768655426 +0000 UTC m=+715.904379981" watchObservedRunningTime="2026-07-06 22:58:43.770000392 +0000 UTC m=+715.905724946" Jul 06 22:58:44.584703 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:44.584665 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:58:52.506000 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:52.505950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:52.506447 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:52.506030 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:52.518254 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:52.518230 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:52.806456 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:52.806373 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9" Jul 06 22:58:54.022073 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:58:54.022022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:22.290379 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:59:22.290348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:22.386746 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:59:22.386715 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:23.386638 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:59:23.386604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:23.480107 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:59:23.480068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 22:59:50.476837 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:59:50.476801 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 22:59:59.682715 ip-10-0-134-52 kubenswrapper[2579]: I0706 22:59:59.682680 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:00:12.080743 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:12.080662 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:00:19.783523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:19.778038 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:00:31.081398 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:31.081362 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:00:32.806068 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:32.806040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:34.042780 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:34.042731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:34.979150 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:34.979101 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:36.817179 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:36.817137 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:49.804486 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:49.804451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:49.860912 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:00:49.860874 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:00.180091 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:00.180056 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:01:13.379185 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:13.379140 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:01:17.801069 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:17.801001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:17.898397 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:17.898361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:18.001091 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:18.001062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:23.764861 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:23.764819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:23.872101 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:23.872071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:23.967367 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:23.967335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:24.094604 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:24.094515 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:29.716175 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:29.716138 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:29.816495 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:29.816463 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:45.281044 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:45.281008 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:01:48.327920 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:48.327889 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:01:48.330421 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:48.330391 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:01:48.595783 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:48.595690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:54.478788 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:01:54.478754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:02:03.388240 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:02:03.388204 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:02:08.878289 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:02:08.878253 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:02:20.170789 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:02:20.170751 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:02:29.379423 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:02:29.379389 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:02:40.773895 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:02:40.773859 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:02:50.178417 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:02:50.178375 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:03:06.556282 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:03:06.556245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:12.986014 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:03:12.985928 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:03:24.080587 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:03:24.080542 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:03:47.477500 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:03:47.477459 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:03:58.087348 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:03:58.087314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:04:12.470012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:04:12.469974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:04:22.977703 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:04:22.977668 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:04:32.585962 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:04:32.585930 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:04:42.481784 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:04:42.481692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:04:52.381145 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:04:52.381089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:06:41.685489 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:06:41.685446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:06:48.361857 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:06:48.361828 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:06:48.362293 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:06:48.361995 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:06:50.879076 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:06:50.879043 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:07:05.182965 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:07:05.182931 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:07:17.188559 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:07:17.188522 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:07:28.880109 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:07:28.880070 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:07:38.383604 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:07:38.383565 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:07:48.278563 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:07:48.278486 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:07:57.873523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:07:57.873486 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:08:05.199731 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:08:05.199697 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:08:15.077208 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:08:15.077177 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:08:24.484961 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:08:24.484924 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:08:30.479354 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:08:30.479320 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:08:53.178854 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:08:53.178806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:09:01.880152 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:01.880107 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:09:11.179673 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:11.179593 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:09:14.878115 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:14.878083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:09:16.256372 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:16.256341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:09:23.084817 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:23.084783 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:09:37.180877 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:37.180838 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:09:48.676803 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:48.676771 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:09:57.383654 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:09:57.383626 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:10:06.983636 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:10:06.983599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:10:07.596245 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:10:07.596217 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:10:16.479968 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:10:16.479936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:10:25.382737 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:10:25.382701 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:10:38.484777 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:10:38.484745 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:10:48.082799 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:10:48.082720 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:10:57.881577 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:10:57.881543 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:11:09.878377 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:09.878342 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:11:20.776546 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:20.776511 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:11:29.097315 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:29.097275 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:11:36.387044 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:36.387012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:11:41.378288 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:41.378254 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:11:48.392466 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:48.392435 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:11:48.394356 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:48.394333 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:11:52.782927 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:11:52.782886 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:12:03.983183 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:03.983148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:12:05.594143 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:05.594080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:13.030110 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:13.030020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:16.577910 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:16.577880 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:12:21.277602 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:21.277567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:25.885859 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:25.885824 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:12:26.369347 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:26.369311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:29.103622 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:29.103590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:29.199528 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:29.199488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:29.536476 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:29.536443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:35.341467 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:35.341430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:35.381208 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:35.381176 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:12:35.443665 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:35.443634 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:35.546071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:35.546041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:35.605622 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:35.605533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:36.732018 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:36.731984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:36.831382 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:36.831351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:36.900474 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:36.900435 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:37.036432 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:37.036324 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:37.281360 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:37.281322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:37.379065 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:37.378980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:39.175652 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:39.175622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:41.716884 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:41.716843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:41.913083 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:41.913024 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:42.012712 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:42.012626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:42.113260 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:42.113232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:43.230552 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:43.230520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:47.721734 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:47.721695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:47.821894 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:47.821865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:47.916969 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:47.916941 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:47.978096 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:47.978022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:49.020011 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:49.019972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:49.108969 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:49.108930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:49.174802 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:49.174762 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:49.555569 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:49.555541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:49.657221 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:49.657195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:49.904785 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:49.904744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:51.187256 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:51.187226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:52.085307 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:52.085274 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:12:54.110309 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:54.110276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:12:54.396410 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:12:54.396369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:00.194685 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:00.194653 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:00.296090 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:00.296058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:03.119798 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:03.119763 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:13:06.035610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:06.035581 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:06.138455 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:06.138423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:06.239028 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:06.238999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:07.337152 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:07.337104 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:07.429882 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:07.429849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:07.521801 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:07.521772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:08.018287 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:08.018255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:08.109362 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:08.109329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:08.179737 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:08.179706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:11.045239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:11.045205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:11.173838 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:11.173806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:13:11.769331 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:11.769293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:11.968808 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:11.968767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:12.193737 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:12.193705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:12.287702 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:12.287673 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:13.560246 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:13.560216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:17.841588 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:17.841551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:17.942051 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:17.942017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:18.421854 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:18.421821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:18.500311 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:18.500282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:19.916692 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:19.916664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:22.183623 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:22.183585 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:13:25.621672 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:25.621639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:25.729180 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:25.729147 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:25.822401 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:25.822360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:25.922328 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:25.922298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:26.356800 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:26.356721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:52.329838 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:52.329806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:57.938011 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:57.937977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:58.004306 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:58.004276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:58.094708 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:58.094674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:13:58.190510 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:13:58.190430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:13.990053 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:13.990018 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:19.569021 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:19.568981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:20.657551 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:20.657516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:30.299822 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:30.299788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:31.077545 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:31.077510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:36.904553 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:36.904514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:36.967993 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:36.967962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:37.107259 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:37.107222 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:37.222059 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:37.221980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:37.937290 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:37.937252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:38.037306 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:38.037272 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:38.138330 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:38.138294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:39.699008 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:39.698980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:40.278802 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:40.278768 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:14:45.274990 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:45.274954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:45.376099 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:45.376069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:45.935511 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:45.935481 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:49.082538 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:49.082501 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:14:51.706259 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:51.706224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:51.767727 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:51.767688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:51.937460 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:51.937428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:52.258299 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:52.258268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:52.423184 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:52.423153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:55.576257 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:55.576224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:55.665973 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:55.665937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:55.731318 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:55.731287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:55.862067 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:55.861986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:56.133882 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:56.133797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:56.221317 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:56.221280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:14:59.775554 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:14:59.775517 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:15:00.077505 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:00.077422 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:00.735385 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:00.735355 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:00.835662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:00.835603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:01.154451 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:01.154414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:06.981947 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:06.981914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:07.047705 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:07.047674 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:07.498749 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:07.498719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:07.814091 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:07.814009 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:07.948239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:07.948202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:13.774913 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:13.774810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:13.872995 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:13.872961 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:13.982959 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:13.982921 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:15:14.029668 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:14.029586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:14.387256 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:14.387171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:14.487047 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:14.486996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:14.586195 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:14.586159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:20.206507 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:20.206466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:20.269048 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:20.269014 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:20.337640 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:20.337610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:20.438102 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:20.438060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:20.547495 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:20.547413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:26.523213 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:26.523176 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:26.623859 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:26.623825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:33.234988 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:33.234952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:33.529808 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:33.529730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:33.669618 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:33.669579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:36.933188 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:36.933150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:37.031234 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:37.031196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:37.117742 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:37.117707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:37.411599 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:37.411565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:37.518799 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:37.518761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:37.651636 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:37.651602 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:41.347598 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:41.347564 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:43.591312 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:43.591278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:15:49.579247 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.579215 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7"] Jul 06 23:15:49.583021 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.582999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.585635 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.585618 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2fa3353e\"/\"default-dockercfg-9wf88\"" Jul 06 23:15:49.586742 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.586725 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2fa3353e\"/\"openshift-service-ca.crt\"" Jul 06 23:15:49.586834 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.586748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2fa3353e\"/\"e2e-lifecycle-model-2fa3353e-kserve-self-signed-certs\"" Jul 06 23:15:49.586884 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.586831 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2fa3353e\"/\"kube-root-ca.crt\"" Jul 06 23:15:49.596569 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.596507 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7"] Jul 06 23:15:49.616039 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.616013 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:49.732269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.732244 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tmp-dir\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.732450 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.732275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-home\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.732450 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.732317 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-dshm\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.732450 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.732357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-model-cache\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.732450 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.732393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnzc\" (UniqueName: \"kubernetes.io/projected/71f08ada-17a9-4f0e-bff2-1a05b97325a0-kube-api-access-rmnzc\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.732593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.732470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tls-certs\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.833639 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.833555 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-dshm\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.833639 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.833594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-model-cache\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.833639 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.833625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnzc\" (UniqueName: \"kubernetes.io/projected/71f08ada-17a9-4f0e-bff2-1a05b97325a0-kube-api-access-rmnzc\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.833926 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.833647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tls-certs\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.833926 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.833689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tmp-dir\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.833926 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.833707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-home\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.834063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.834040 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-model-cache\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.834104 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.834082 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-home\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.834172 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.834098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tmp-dir\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.835813 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.835793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-dshm\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.836146 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.836114 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tls-certs\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.841504 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.841480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnzc\" (UniqueName: \"kubernetes.io/projected/71f08ada-17a9-4f0e-bff2-1a05b97325a0-kube-api-access-rmnzc\") pod \"e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:49.892560 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:49.892534 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:50.022781 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:50.022754 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7"] Jul 06 23:15:50.024438 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:15:50.024405 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71f08ada_17a9_4f0e_bff2_1a05b97325a0.slice/crio-2e40e976954f309c99fd9eee7d20dbe37b5214343cf18e6b7a85d5eb22c97044 WatchSource:0}: Error finding container 2e40e976954f309c99fd9eee7d20dbe37b5214343cf18e6b7a85d5eb22c97044: Status 404 returned error can't find the container with id 2e40e976954f309c99fd9eee7d20dbe37b5214343cf18e6b7a85d5eb22c97044 Jul 06 23:15:50.715010 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:50.714975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" event={"ID":"71f08ada-17a9-4f0e-bff2-1a05b97325a0","Type":"ContainerStarted","Data":"a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017"} Jul 06 23:15:50.715010 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:50.715013 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" event={"ID":"71f08ada-17a9-4f0e-bff2-1a05b97325a0","Type":"ContainerStarted","Data":"2e40e976954f309c99fd9eee7d20dbe37b5214343cf18e6b7a85d5eb22c97044"} Jul 06 23:15:50.733293 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:50.733246 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" podStartSLOduration=1.733231373 podStartE2EDuration="1.733231373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:15:50.731209599 +0000 UTC m=+1742.866934180" watchObservedRunningTime="2026-07-06 23:15:50.733231373 +0000 UTC m=+1742.868955964" Jul 06 23:15:54.588951 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:54.588915 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:15:58.742940 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:58.742908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:58.842339 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:58.842311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:59.442317 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:59.442261 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:59.545262 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:59.545218 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:15:59.892967 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:59.892939 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:59.893433 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:59.893016 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:15:59.904979 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:15:59.904960 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:16:00.774369 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:00.774341 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:16:07.380818 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:07.380776 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:16:13.665759 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:13.665682 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:16:14.054201 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:14.054060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:16:17.975075 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:17.975041 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:16:19.661982 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:19.661936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:16:24.649881 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:24.649851 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7"] Jul 06 23:16:24.698187 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:24.698154 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:16:27.482314 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:27.482278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:16:39.670554 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.670487 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" podUID="71f08ada-17a9-4f0e-bff2-1a05b97325a0" containerName="main" containerID="cri-o://a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017" gracePeriod=45 Jul 06 23:16:39.920285 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.920261 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:16:39.953947 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.953861 2579 generic.go:356] "Generic (PLEG): container finished" podID="71f08ada-17a9-4f0e-bff2-1a05b97325a0" containerID="a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017" exitCode=0 Jul 06 23:16:39.953947 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.953924 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" Jul 06 23:16:39.954181 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.953950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" event={"ID":"71f08ada-17a9-4f0e-bff2-1a05b97325a0","Type":"ContainerDied","Data":"a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017"} Jul 06 23:16:39.954181 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.953993 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7" event={"ID":"71f08ada-17a9-4f0e-bff2-1a05b97325a0","Type":"ContainerDied","Data":"2e40e976954f309c99fd9eee7d20dbe37b5214343cf18e6b7a85d5eb22c97044"} Jul 06 23:16:39.954181 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.954015 2579 scope.go:117] "RemoveContainer" containerID="a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017" Jul 06 23:16:39.963350 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.963330 2579 scope.go:117] "RemoveContainer" containerID="a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017" Jul 06 23:16:39.963632 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:16:39.963614 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017\": container with ID starting with a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017 not found: ID does not exist" containerID="a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017" Jul 06 23:16:39.963718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:39.963639 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017"} err="failed to get container status \"a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017\": rpc error: code = NotFound desc = could not find container \"a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017\": container with ID starting with a29bd5df0bae8dc1b6968bcb392b92b642703eb8f6d2511d8d9bf26882445017 not found: ID does not exist" Jul 06 23:16:40.004629 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.004604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-model-cache\") pod \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " Jul 06 23:16:40.004737 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.004696 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmnzc\" (UniqueName: \"kubernetes.io/projected/71f08ada-17a9-4f0e-bff2-1a05b97325a0-kube-api-access-rmnzc\") pod \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " Jul 06 23:16:40.004794 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.004768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-home\") pod \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " Jul 06 23:16:40.004856 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.004835 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tmp-dir\") pod \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " Jul 06 23:16:40.004912 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.004861 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tls-certs\") pod \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " Jul 06 23:16:40.004912 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.004878 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "71f08ada-17a9-4f0e-bff2-1a05b97325a0" (UID: "71f08ada-17a9-4f0e-bff2-1a05b97325a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:16:40.004912 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.004887 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-dshm\") pod \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\" (UID: \"71f08ada-17a9-4f0e-bff2-1a05b97325a0\") " Jul 06 23:16:40.005064 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.005046 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-home" (OuterVolumeSpecName: "home") pod "71f08ada-17a9-4f0e-bff2-1a05b97325a0" (UID: "71f08ada-17a9-4f0e-bff2-1a05b97325a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:16:40.005136 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.005057 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "71f08ada-17a9-4f0e-bff2-1a05b97325a0" (UID: "71f08ada-17a9-4f0e-bff2-1a05b97325a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:16:40.005305 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.005283 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:16:40.005305 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.005303 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:16:40.005468 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.005312 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:16:40.006925 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.006897 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71f08ada-17a9-4f0e-bff2-1a05b97325a0" (UID: "71f08ada-17a9-4f0e-bff2-1a05b97325a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:16:40.007014 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.006937 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f08ada-17a9-4f0e-bff2-1a05b97325a0-kube-api-access-rmnzc" (OuterVolumeSpecName: "kube-api-access-rmnzc") pod "71f08ada-17a9-4f0e-bff2-1a05b97325a0" (UID: "71f08ada-17a9-4f0e-bff2-1a05b97325a0"). InnerVolumeSpecName "kube-api-access-rmnzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:16:40.007014 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.006935 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-dshm" (OuterVolumeSpecName: "dshm") pod "71f08ada-17a9-4f0e-bff2-1a05b97325a0" (UID: "71f08ada-17a9-4f0e-bff2-1a05b97325a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:16:40.106289 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.106252 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71f08ada-17a9-4f0e-bff2-1a05b97325a0-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:16:40.106289 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.106279 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/71f08ada-17a9-4f0e-bff2-1a05b97325a0-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:16:40.106289 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.106289 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmnzc\" (UniqueName: \"kubernetes.io/projected/71f08ada-17a9-4f0e-bff2-1a05b97325a0-kube-api-access-rmnzc\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:16:40.275467 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.275438 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7"] Jul 06 23:16:40.281251 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.281226 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2fa3353e/e2e-lifecycle-model-2fa3353e-kserve-776cdd464c-z7wf7"] Jul 06 23:16:40.427776 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:40.427748 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71f08ada-17a9-4f0e-bff2-1a05b97325a0" path="/var/lib/kubelet/pods/71f08ada-17a9-4f0e-bff2-1a05b97325a0/volumes" Jul 06 23:16:48.430969 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:48.430939 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:16:48.435289 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:16:48.435268 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:17:01.156549 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:01.156507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:01.671268 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:01.671242 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:03.781784 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:03.781753 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:17:07.257141 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:07.257093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:13.363389 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:13.363352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:13.464399 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:13.464364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:29.984033 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:29.983999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:30.086284 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:30.086248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:30.272002 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:30.271918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:31.550340 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:31.550304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:31.647409 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:31.647381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:31.743986 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:31.743951 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:32.428911 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:32.428876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:32.529390 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:32.529353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:32.631045 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:32.631011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:33.815512 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:33.815480 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:17:33.988458 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:33.988416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:39.780844 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:39.780810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:39.881737 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:39.881703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:40.456189 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:40.456156 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:46.288472 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:46.288439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:46.360883 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:46.360845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:46.446871 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:46.446834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:46.553993 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:46.553919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:47.678534 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:47.678500 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:17:57.048569 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:57.048534 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:17:57.356366 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:17:57.356273 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:18.537927 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:18.537843 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:24.212536 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:24.212501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:24.313628 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:24.313590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:24.576040 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:24.575956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:24.877234 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:24.877158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:24.977701 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:24.977670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:27.818864 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:27.818832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:27.897640 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:27.897613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:27.960520 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:27.960491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:28.491954 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:28.491916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:28.592748 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:28.592719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:32.319887 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:32.319850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:32.637699 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:32.637617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:32.951220 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:32.951189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:33.555084 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:33.555054 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:35.491090 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:35.491053 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:18:35.879370 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:35.879290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:35.950294 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:35.950263 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:36.017477 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:36.017450 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:36.122249 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:36.122216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:36.436499 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:36.436466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:36.537759 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:36.537730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:40.417354 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:40.417319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:45.776074 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:45.776041 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:48.717275 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.717244 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z"] Jul 06 23:18:48.717722 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.717648 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71f08ada-17a9-4f0e-bff2-1a05b97325a0" containerName="main" Jul 06 23:18:48.717722 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.717659 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="71f08ada-17a9-4f0e-bff2-1a05b97325a0" containerName="main" Jul 06 23:18:48.717802 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.717740 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="71f08ada-17a9-4f0e-bff2-1a05b97325a0" containerName="main" Jul 06 23:18:48.720780 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.720763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.723236 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.723213 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2ff615\"/\"openshift-service-ca.crt\"" Jul 06 23:18:48.724183 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.724162 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2ff615\"/\"test-model-2ff615-kserve-self-signed-certs\"" Jul 06 23:18:48.724273 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.724185 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2ff615\"/\"default-dockercfg-ch2j9\"" Jul 06 23:18:48.724273 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.724195 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2ff615\"/\"kube-root-ca.crt\"" Jul 06 23:18:48.727776 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.727755 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z"] Jul 06 23:18:48.808321 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.808279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-dshm\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.808515 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.808325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-model-cache\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.808515 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.808352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-home\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.808515 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.808454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79fr\" (UniqueName: \"kubernetes.io/projected/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-kube-api-access-v79fr\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.808656 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.808517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tmp-dir\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.808656 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.808576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tls-certs\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.875261 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.875225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:48.909303 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tmp-dir\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tls-certs\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909375 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-dshm\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-model-cache\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-home\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79fr\" (UniqueName: \"kubernetes.io/projected/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-kube-api-access-v79fr\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909861 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tmp-dir\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909861 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-home\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.909961 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.909862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-model-cache\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.911696 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.911678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-dshm\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.911983 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.911962 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tls-certs\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:48.917087 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:48.917066 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79fr\" (UniqueName: \"kubernetes.io/projected/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-kube-api-access-v79fr\") pod \"test-model-2ff615-kserve-6984c859b5-rhx7z\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:49.031420 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:49.031326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:49.169614 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:49.169590 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z"] Jul 06 23:18:49.171585 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:18:49.171560 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9918ec0b_7ab3_45c5_84ff_32dfb4da4368.slice/crio-4c3076eebd588e290684d72513e3b9666fc28287e7103b05af66475a3b1702e4 WatchSource:0}: Error finding container 4c3076eebd588e290684d72513e3b9666fc28287e7103b05af66475a3b1702e4: Status 404 returned error can't find the container with id 4c3076eebd588e290684d72513e3b9666fc28287e7103b05af66475a3b1702e4 Jul 06 23:18:49.585776 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:49.585742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" event={"ID":"9918ec0b-7ab3-45c5-84ff-32dfb4da4368","Type":"ContainerStarted","Data":"e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd"} Jul 06 23:18:49.585776 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:49.585780 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" event={"ID":"9918ec0b-7ab3-45c5-84ff-32dfb4da4368","Type":"ContainerStarted","Data":"4c3076eebd588e290684d72513e3b9666fc28287e7103b05af66475a3b1702e4"} Jul 06 23:18:49.603932 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:49.603880 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" podStartSLOduration=1.603862147 podStartE2EDuration="1.603862147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:18:49.601642861 +0000 UTC m=+1921.737367417" watchObservedRunningTime="2026-07-06 23:18:49.603862147 +0000 UTC m=+1921.739586702" Jul 06 23:18:49.793549 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:49.793512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:18:49.988698 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:49.988662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:50.089922 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:50.089892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:58.061396 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:58.061357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:58.161541 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:58.161504 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:18:59.032145 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:59.032085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:59.032383 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:59.032172 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:59.044814 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:59.044788 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:18:59.647902 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:18:59.647874 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:19:02.479523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:02.479436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:19:16.287709 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.287675 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b"] Jul 06 23:19:16.291256 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.291238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.294611 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.294543 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f9d5b5\"/\"kube-root-ca.crt\"" Jul 06 23:19:16.294611 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.294570 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f9d5b5\"/\"default-dockercfg-9kht9\"" Jul 06 23:19:16.294611 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.294568 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f9d5b5\"/\"openshift-service-ca.crt\"" Jul 06 23:19:16.294911 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.294584 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f9d5b5\"/\"test-model-f9d5b5-kserve-self-signed-certs\"" Jul 06 23:19:16.300643 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.300620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b"] Jul 06 23:19:16.369598 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.369564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-dshm\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.369767 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.369603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50179846-7cbf-4486-980f-300cad44beaa-tls-certs\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.369767 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.369684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-home\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.369767 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.369729 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-model-cache\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.369767 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.369761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-tmp-dir\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.369907 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.369805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shxff\" (UniqueName: \"kubernetes.io/projected/50179846-7cbf-4486-980f-300cad44beaa-kube-api-access-shxff\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.387207 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.387150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:16.471186 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-home\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471186 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-model-cache\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471447 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-tmp-dir\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471447 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shxff\" (UniqueName: \"kubernetes.io/projected/50179846-7cbf-4486-980f-300cad44beaa-kube-api-access-shxff\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471447 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-dshm\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471447 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50179846-7cbf-4486-980f-300cad44beaa-tls-certs\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471658 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471602 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-home\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471714 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-model-cache\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.471714 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.471680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-tmp-dir\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.473540 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.473518 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-dshm\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.473869 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.473852 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50179846-7cbf-4486-980f-300cad44beaa-tls-certs\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.480033 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.480001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shxff\" (UniqueName: \"kubernetes.io/projected/50179846-7cbf-4486-980f-300cad44beaa-kube-api-access-shxff\") pod \"test-model-f9d5b5-kserve-655d5496d7-jbr7b\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.602476 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.602374 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:16.742581 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:16.742557 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b"] Jul 06 23:19:16.744592 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:19:16.744567 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50179846_7cbf_4486_980f_300cad44beaa.slice/crio-fae49f6211251909ce4e5aa87f46cc6a2e77f5278a9051aae9a85ed54f9dac01 WatchSource:0}: Error finding container fae49f6211251909ce4e5aa87f46cc6a2e77f5278a9051aae9a85ed54f9dac01: Status 404 returned error can't find the container with id fae49f6211251909ce4e5aa87f46cc6a2e77f5278a9051aae9a85ed54f9dac01 Jul 06 23:19:17.731812 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:17.731762 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" event={"ID":"50179846-7cbf-4486-980f-300cad44beaa","Type":"ContainerStarted","Data":"9ca32798b5f09bdd541c74e2c7eec9b5c45f31ccdc0f4d3b747303aba5c648b1"} Jul 06 23:19:17.731812 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:17.731810 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" event={"ID":"50179846-7cbf-4486-980f-300cad44beaa","Type":"ContainerStarted","Data":"fae49f6211251909ce4e5aa87f46cc6a2e77f5278a9051aae9a85ed54f9dac01"} Jul 06 23:19:17.749726 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:17.749668 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" podStartSLOduration=1.749652861 podStartE2EDuration="1.749652861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:19:17.748862278 +0000 UTC m=+1949.884586833" watchObservedRunningTime="2026-07-06 23:19:17.749652861 +0000 UTC m=+1949.885377416" Jul 06 23:19:21.590425 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:21.590383 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:19:26.603529 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:26.603484 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:26.603994 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:26.603553 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:26.616027 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:26.615997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:26.792097 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:26.792070 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:19:43.817154 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:43.817055 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z"] Jul 06 23:19:43.853453 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:43.853410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:44.808645 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:44.808613 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b"] Jul 06 23:19:45.033446 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:45.033415 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:45.133444 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:45.133360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:45.234836 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:45.234791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:45.532485 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:45.532442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:45.633299 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:45.633264 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:45.733257 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:45.733224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:45.832901 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:45.832810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:48.476982 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:48.476943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:19:50.603867 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:50.603813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:19:58.837992 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:58.837922 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" podUID="9918ec0b-7ab3-45c5-84ff-32dfb4da4368" containerName="main" containerID="cri-o://e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd" gracePeriod=45 Jul 06 23:19:59.085282 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.085258 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:19:59.192241 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192193 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-model-cache\") pod \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " Jul 06 23:19:59.192465 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192258 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v79fr\" (UniqueName: \"kubernetes.io/projected/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-kube-api-access-v79fr\") pod \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " Jul 06 23:19:59.192465 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192300 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-home\") pod \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " Jul 06 23:19:59.192465 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192336 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tmp-dir\") pod \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " Jul 06 23:19:59.192465 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192418 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-dshm\") pod \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " Jul 06 23:19:59.192682 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192466 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tls-certs\") pod \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\" (UID: \"9918ec0b-7ab3-45c5-84ff-32dfb4da4368\") " Jul 06 23:19:59.192682 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192567 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-model-cache" (OuterVolumeSpecName: "model-cache") pod "9918ec0b-7ab3-45c5-84ff-32dfb4da4368" (UID: "9918ec0b-7ab3-45c5-84ff-32dfb4da4368"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:19:59.192682 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192575 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9918ec0b-7ab3-45c5-84ff-32dfb4da4368" (UID: "9918ec0b-7ab3-45c5-84ff-32dfb4da4368"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:19:59.192682 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192661 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-home" (OuterVolumeSpecName: "home") pod "9918ec0b-7ab3-45c5-84ff-32dfb4da4368" (UID: "9918ec0b-7ab3-45c5-84ff-32dfb4da4368"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:19:59.192876 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192859 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:19:59.192876 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192875 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:19:59.192969 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.192883 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:19:59.194625 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.194591 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9918ec0b-7ab3-45c5-84ff-32dfb4da4368" (UID: "9918ec0b-7ab3-45c5-84ff-32dfb4da4368"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:19:59.194739 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.194623 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-dshm" (OuterVolumeSpecName: "dshm") pod "9918ec0b-7ab3-45c5-84ff-32dfb4da4368" (UID: "9918ec0b-7ab3-45c5-84ff-32dfb4da4368"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:19:59.194739 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.194651 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-kube-api-access-v79fr" (OuterVolumeSpecName: "kube-api-access-v79fr") pod "9918ec0b-7ab3-45c5-84ff-32dfb4da4368" (UID: "9918ec0b-7ab3-45c5-84ff-32dfb4da4368"). InnerVolumeSpecName "kube-api-access-v79fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:19:59.293957 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.293916 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:19:59.293957 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.293948 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v79fr\" (UniqueName: \"kubernetes.io/projected/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-kube-api-access-v79fr\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:19:59.293957 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.293961 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9918ec0b-7ab3-45c5-84ff-32dfb4da4368-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:19:59.830009 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.829945 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" podUID="50179846-7cbf-4486-980f-300cad44beaa" containerName="main" containerID="cri-o://9ca32798b5f09bdd541c74e2c7eec9b5c45f31ccdc0f4d3b747303aba5c648b1" gracePeriod=45 Jul 06 23:19:59.953670 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.953635 2579 generic.go:356] "Generic (PLEG): container finished" podID="9918ec0b-7ab3-45c5-84ff-32dfb4da4368" containerID="e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd" exitCode=0 Jul 06 23:19:59.953670 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.953668 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" Jul 06 23:19:59.954211 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.953709 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" event={"ID":"9918ec0b-7ab3-45c5-84ff-32dfb4da4368","Type":"ContainerDied","Data":"e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd"} Jul 06 23:19:59.954211 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.953766 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z" event={"ID":"9918ec0b-7ab3-45c5-84ff-32dfb4da4368","Type":"ContainerDied","Data":"4c3076eebd588e290684d72513e3b9666fc28287e7103b05af66475a3b1702e4"} Jul 06 23:19:59.954211 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.953790 2579 scope.go:117] "RemoveContainer" containerID="e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd" Jul 06 23:19:59.955742 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.955721 2579 generic.go:356] "Generic (PLEG): container finished" podID="50179846-7cbf-4486-980f-300cad44beaa" containerID="9ca32798b5f09bdd541c74e2c7eec9b5c45f31ccdc0f4d3b747303aba5c648b1" exitCode=0 Jul 06 23:19:59.955838 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.955752 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" event={"ID":"50179846-7cbf-4486-980f-300cad44beaa","Type":"ContainerDied","Data":"9ca32798b5f09bdd541c74e2c7eec9b5c45f31ccdc0f4d3b747303aba5c648b1"} Jul 06 23:19:59.964690 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.964671 2579 scope.go:117] "RemoveContainer" containerID="e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd" Jul 06 23:19:59.965033 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:19:59.965015 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd\": container with ID starting with e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd not found: ID does not exist" containerID="e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd" Jul 06 23:19:59.965111 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.965044 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd"} err="failed to get container status \"e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd\": rpc error: code = NotFound desc = could not find container \"e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd\": container with ID starting with e4287b6bd1e0ca3d80f693844b1d0c6003b3789d358d71076f4921752e993fcd not found: ID does not exist" Jul 06 23:19:59.979812 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.979783 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z"] Jul 06 23:19:59.984564 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:19:59.984538 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2ff615/test-model-2ff615-kserve-6984c859b5-rhx7z"] Jul 06 23:20:00.089868 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.089808 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:20:00.203385 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203352 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-tmp-dir\") pod \"50179846-7cbf-4486-980f-300cad44beaa\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " Jul 06 23:20:00.203588 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203412 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-home\") pod \"50179846-7cbf-4486-980f-300cad44beaa\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " Jul 06 23:20:00.203588 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203434 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-model-cache\") pod \"50179846-7cbf-4486-980f-300cad44beaa\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " Jul 06 23:20:00.203588 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203456 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50179846-7cbf-4486-980f-300cad44beaa-tls-certs\") pod \"50179846-7cbf-4486-980f-300cad44beaa\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " Jul 06 23:20:00.203588 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203508 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-dshm\") pod \"50179846-7cbf-4486-980f-300cad44beaa\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " Jul 06 23:20:00.203588 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203547 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shxff\" (UniqueName: \"kubernetes.io/projected/50179846-7cbf-4486-980f-300cad44beaa-kube-api-access-shxff\") pod \"50179846-7cbf-4486-980f-300cad44beaa\" (UID: \"50179846-7cbf-4486-980f-300cad44beaa\") " Jul 06 23:20:00.203850 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203668 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50179846-7cbf-4486-980f-300cad44beaa" (UID: "50179846-7cbf-4486-980f-300cad44beaa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:20:00.203850 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203713 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-model-cache" (OuterVolumeSpecName: "model-cache") pod "50179846-7cbf-4486-980f-300cad44beaa" (UID: "50179846-7cbf-4486-980f-300cad44beaa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:20:00.203850 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203728 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-home" (OuterVolumeSpecName: "home") pod "50179846-7cbf-4486-980f-300cad44beaa" (UID: "50179846-7cbf-4486-980f-300cad44beaa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:20:00.203978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203852 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:20:00.203978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203871 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:20:00.203978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.203885 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:20:00.205788 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.205766 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-dshm" (OuterVolumeSpecName: "dshm") pod "50179846-7cbf-4486-980f-300cad44beaa" (UID: "50179846-7cbf-4486-980f-300cad44beaa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:20:00.205897 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.205806 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50179846-7cbf-4486-980f-300cad44beaa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50179846-7cbf-4486-980f-300cad44beaa" (UID: "50179846-7cbf-4486-980f-300cad44beaa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:20:00.205897 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.205852 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50179846-7cbf-4486-980f-300cad44beaa-kube-api-access-shxff" (OuterVolumeSpecName: "kube-api-access-shxff") pod "50179846-7cbf-4486-980f-300cad44beaa" (UID: "50179846-7cbf-4486-980f-300cad44beaa"). InnerVolumeSpecName "kube-api-access-shxff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:20:00.305236 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.305201 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50179846-7cbf-4486-980f-300cad44beaa-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:20:00.305236 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.305229 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shxff\" (UniqueName: \"kubernetes.io/projected/50179846-7cbf-4486-980f-300cad44beaa-kube-api-access-shxff\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:20:00.305236 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.305239 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50179846-7cbf-4486-980f-300cad44beaa-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:20:00.428690 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.428649 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9918ec0b-7ab3-45c5-84ff-32dfb4da4368" path="/var/lib/kubelet/pods/9918ec0b-7ab3-45c5-84ff-32dfb4da4368/volumes" Jul 06 23:20:00.961745 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.961717 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" Jul 06 23:20:00.962227 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.961759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b" event={"ID":"50179846-7cbf-4486-980f-300cad44beaa","Type":"ContainerDied","Data":"fae49f6211251909ce4e5aa87f46cc6a2e77f5278a9051aae9a85ed54f9dac01"} Jul 06 23:20:00.962227 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.961816 2579 scope.go:117] "RemoveContainer" containerID="9ca32798b5f09bdd541c74e2c7eec9b5c45f31ccdc0f4d3b747303aba5c648b1" Jul 06 23:20:00.984568 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.984534 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b"] Jul 06 23:20:00.988974 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:00.988933 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f9d5b5/test-model-f9d5b5-kserve-655d5496d7-jbr7b"] Jul 06 23:20:02.427816 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:02.427785 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50179846-7cbf-4486-980f-300cad44beaa" path="/var/lib/kubelet/pods/50179846-7cbf-4486-980f-300cad44beaa/volumes" Jul 06 23:20:27.466181 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:27.466106 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:27.735105 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:27.734975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:28.156938 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:28.156896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:28.581151 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:28.581004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:28.679718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:28.679690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:28.776637 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:28.776610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:28.855187 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:28.855071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:31.062502 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:31.062457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:31.484349 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:31.484307 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:20:34.561501 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:34.561468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:34.662983 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:34.662949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:34.909619 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:34.909586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:35.394653 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:35.394623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:35.698360 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:35.698277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:36.017625 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:36.017547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:38.648045 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:38.648007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:38.730609 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:38.730580 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:38.799409 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:38.799372 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:38.898309 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:38.898202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:39.475483 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:39.475451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:39.576109 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:39.576074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:43.152668 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:43.152635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:43.589579 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:43.589496 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:43.955998 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:43.955963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:44.083904 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:44.083852 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:45.688364 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:45.688329 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:20:46.681078 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:46.681015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:46.771073 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:46.771035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:46.857102 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:46.857062 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:47.463668 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:47.463636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:47.563634 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:47.563608 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:47.879769 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:47.879680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:51.174350 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:51.174309 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:59.177669 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:59.177633 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:59.474333 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:59.474249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:59.620642 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:59.620604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:20:59.769022 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:20:59.768935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:02.089261 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:02.089218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:21:05.350799 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:05.350759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:05.450767 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:05.450731 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:05.694492 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:05.694456 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:06.012163 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:06.012047 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:06.131823 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:06.131792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:06.232043 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:06.232003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:06.396787 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:06.396750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:06.497666 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:06.497632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:06.601426 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:06.601389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:06.813620 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:06.813543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:12.651230 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:12.651143 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:12.753977 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:12.753945 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:12.923614 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:12.923523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:13.464646 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:13.464613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:13.781879 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:13.781801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:13.881442 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:13.881405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:15.779533 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:15.779490 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:21:16.883794 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:16.883765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:16.983319 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:16.983286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:17.123268 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:17.123238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:17.415504 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:17.415469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:17.615556 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:17.615517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:17.715993 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:17.715907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:21.351134 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:21.351080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:21.674107 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:21.674067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:21.988050 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:21.987968 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:22.309636 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:22.309543 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:25.007992 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:25.007963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:25.108082 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:25.108050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:25.225907 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:25.225867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:25.322381 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:25.322300 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:25.756706 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:25.756671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:25.857065 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:25.857012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:26.111618 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:26.111521 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:21:29.474671 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:29.474640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:36.288246 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:36.288211 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:21:38.622242 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:38.622204 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:38.718036 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:38.717997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:41.272305 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.272261 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246"] Jul 06 23:21:41.272931 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.272898 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9918ec0b-7ab3-45c5-84ff-32dfb4da4368" containerName="main" Jul 06 23:21:41.272931 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.272923 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9918ec0b-7ab3-45c5-84ff-32dfb4da4368" containerName="main" Jul 06 23:21:41.273045 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.272936 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50179846-7cbf-4486-980f-300cad44beaa" containerName="main" Jul 06 23:21:41.273045 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.272945 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50179846-7cbf-4486-980f-300cad44beaa" containerName="main" Jul 06 23:21:41.273112 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.273052 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50179846-7cbf-4486-980f-300cad44beaa" containerName="main" Jul 06 23:21:41.273112 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.273067 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9918ec0b-7ab3-45c5-84ff-32dfb4da4368" containerName="main" Jul 06 23:21:41.276786 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.276768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.279521 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.279484 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"kube-root-ca.crt\"" Jul 06 23:21:41.279622 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.279595 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"openshift-service-ca.crt\"" Jul 06 23:21:41.280645 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.280623 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"auth-test-model-2cc96a-kserve-self-signed-certs\"" Jul 06 23:21:41.280745 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.280630 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"default-dockercfg-42l67\"" Jul 06 23:21:41.285052 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.285033 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246"] Jul 06 23:21:41.305692 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.305662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:41.369505 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.369465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-dshm\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.369505 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.369511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tmp-dir\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.369730 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.369528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tls-certs\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.369730 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.369548 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-home\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.369730 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.369564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-model-cache\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.369730 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.369636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr4g9\" (UniqueName: \"kubernetes.io/projected/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-kube-api-access-hr4g9\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470328 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-home\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470328 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-model-cache\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470555 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr4g9\" (UniqueName: \"kubernetes.io/projected/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-kube-api-access-hr4g9\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470555 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-dshm\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470555 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470490 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tmp-dir\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470555 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tls-certs\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470807 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470787 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-model-cache\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470807 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-home\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.470929 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.470855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tmp-dir\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.472852 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.472831 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-dshm\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.473026 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.473009 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tls-certs\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.478317 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.478296 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr4g9\" (UniqueName: \"kubernetes.io/projected/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-kube-api-access-hr4g9\") pod \"auth-test-model-2cc96a-kserve-64b9fb659f-d5246\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.587881 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.587787 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:41.723628 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:41.723602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246"] Jul 06 23:21:41.725605 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:21:41.725574 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b028e4b_87e7_4f5d_a8fb_6d1ee76725b0.slice/crio-681c57789e53cc37f6d6045c0914590fdd8c0e60bd22d2b65e4970e063d08732 WatchSource:0}: Error finding container 681c57789e53cc37f6d6045c0914590fdd8c0e60bd22d2b65e4970e063d08732: Status 404 returned error can't find the container with id 681c57789e53cc37f6d6045c0914590fdd8c0e60bd22d2b65e4970e063d08732 Jul 06 23:21:42.455266 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:42.455228 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" event={"ID":"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0","Type":"ContainerStarted","Data":"c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f"} Jul 06 23:21:42.455266 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:42.455276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" event={"ID":"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0","Type":"ContainerStarted","Data":"681c57789e53cc37f6d6045c0914590fdd8c0e60bd22d2b65e4970e063d08732"} Jul 06 23:21:42.487848 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:42.487800 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" podStartSLOduration=1.487784077 podStartE2EDuration="1.487784077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:21:42.486787141 +0000 UTC m=+2094.622511696" watchObservedRunningTime="2026-07-06 23:21:42.487784077 +0000 UTC m=+2094.623508675" Jul 06 23:21:47.693952 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:47.693915 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:47.796748 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:47.796713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:21:48.461607 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:48.461580 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:21:48.468625 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:48.468592 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:21:51.588239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:51.588200 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:51.588655 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:51.588255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:51.600699 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:51.600676 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:52.517544 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:21:52.517518 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:21:58.869194 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:21:58.869146 2579 controller.go:195] "Failed to update lease" err="Put \"https://a627ca4ccd33443d7a2038d2d51ed17f-941f27fd690b701a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-134-52.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 23:22:08.870002 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:22:08.869955 2579 controller.go:195] "Failed to update lease" err="Put \"https://a627ca4ccd33443d7a2038d2d51ed17f-941f27fd690b701a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-134-52.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 23:22:18.870569 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:22:18.870514 2579 controller.go:195] "Failed to update lease" err="Put \"https://a627ca4ccd33443d7a2038d2d51ed17f-941f27fd690b701a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-134-52.ec2.internal?timeout=10s\": context deadline exceeded" Jul 06 23:22:28.073239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073196 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073203 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-klr96\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073202 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kth7n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073209 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q5276\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073239 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073245 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vmht5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073222 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073262 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073208 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073297 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073274 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073319 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073343 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073365 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073366 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073382 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073391 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073394 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-57bqg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073413 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073424 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.073978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073454 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b6zcw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073497 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073498 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073515 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073535 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073551 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ds5zr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073564 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073573 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073577 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073601 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073612 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073623 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-pc5sr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073660 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073662 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073672 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7xhr7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.074711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073684 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073688 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073697 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073713 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073727 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073732 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073758 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kjn6w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073786 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073793 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-z62jv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073794 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073827 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073828 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073842 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-eujn6c29h5m9p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073849 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073866 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075269 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073893 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073897 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073895 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073899 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073912 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073940 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-snqg8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073952 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073949 2579 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073986 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njcl6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073990 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.073998 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c4v9t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074004 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074017 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074018 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074044 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.075717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074045 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074055 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074064 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074051 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074072 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074075 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074082 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074092 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074093 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vzb54\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074100 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2z7cj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074149 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f7t4d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074174 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074178 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074174 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074172 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-279gpg6osqfm0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074197 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074201 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074216 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9ddx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074223 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074207 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xdg5g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:22:28.074211 2579 controller.go:195] "Failed to update lease" err="Put \"https://a627ca4ccd33443d7a2038d2d51ed17f-941f27fd690b701a.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-134-52.ec2.internal?timeout=10s\": http2: client connection lost" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074251 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074259 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074207 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-86knk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074270 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9fkc5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074219 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074290 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074291 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074306 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074313 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-bs2s8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.076662 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074324 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074328 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074333 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074201 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074340 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cbcdr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074361 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074373 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074385 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074388 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074393 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074401 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074396 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074430 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t96cf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074437 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074433 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074449 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"default-dockercfg-42l67\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074462 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074476 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vwb4x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074483 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074495 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074500 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s88w6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074500 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074526 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074532 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074540 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074554 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074563 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074575 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074578 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9r88k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074586 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.077610 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074595 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7www5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074598 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074625 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074628 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wjzvb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074633 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074624 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074645 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gqm8g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074657 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074659 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074670 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074675 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074684 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074674 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074695 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074708 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078055 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074712 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074707 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074629 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074712 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074693 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074732 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"default-dockercfg-6sq42\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074750 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jmlxb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074755 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074761 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074764 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074779 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39ut8t85kt3o4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074790 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074780 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dx78r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074793 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074812 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.078558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074817 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2zdp4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074806 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"auth-test-model-2cc96a-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074827 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074829 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074834 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074816 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ctk6j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074790 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074849 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074782 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.079012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.074861 2579 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 06 23:22:28.897646 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.897610 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 23:22:28.897646 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.897610 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9fkc5\"" Jul 06 23:22:28.904614 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.904591 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:22:28.908481 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.908466 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 23:22:28.910214 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.910201 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-snqg8\"" Jul 06 23:22:28.911456 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.911443 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"kube-root-ca.crt\"" Jul 06 23:22:28.923369 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.923351 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j9ddx\"" Jul 06 23:22:28.933987 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.933966 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 23:22:28.935220 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.935203 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 06 23:22:28.940095 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.940078 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 23:22:28.941555 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.941533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 23:22:28.950737 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.950714 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gqm8g\"" Jul 06 23:22:28.954440 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.954425 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s88w6\"" Jul 06 23:22:28.955361 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.955348 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xdg5g\"" Jul 06 23:22:28.956109 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.956092 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 23:22:28.963755 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.963740 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-vzb54\"" Jul 06 23:22:28.973481 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.973464 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 23:22:28.975413 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.975399 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vwb4x\"" Jul 06 23:22:28.981345 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.981328 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7www5\"" Jul 06 23:22:28.983279 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.983266 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-klr96\"" Jul 06 23:22:28.984135 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.984112 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vmht5\"" Jul 06 23:22:28.987580 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.987562 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 23:22:28.987660 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.987585 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 23:22:28.992312 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:28.992280 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 23:22:29.004411 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.004384 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2z7cj\"" Jul 06 23:22:29.006356 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.006336 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 06 23:22:29.011193 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.011174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kth7n\"" Jul 06 23:22:29.013096 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.013079 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9r88k\"" Jul 06 23:22:29.016652 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.016634 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 23:22:29.019689 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.019671 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.034798 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.034770 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.035366 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.035351 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 23:22:29.039106 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.039088 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.041679 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.041660 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 23:22:29.056266 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.056215 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 23:22:29.056684 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.056665 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.069038 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.069006 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 23:22:29.080632 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.080605 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 23:22:29.085401 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.085380 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.090229 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.090208 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 23:22:29.092236 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.092217 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 23:22:29.092363 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.092345 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t96cf\"" Jul 06 23:22:29.093708 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.093693 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 23:22:29.097261 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.097243 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 23:22:29.099086 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.099071 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 23:22:29.100237 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.100219 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.102080 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.102063 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.102155 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.102146 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-b6zcw\"" Jul 06 23:22:29.121079 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.121052 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 06 23:22:29.123006 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.122983 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2zdp4\"" Jul 06 23:22:29.129074 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.129048 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 23:22:29.133770 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.133750 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.135568 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.135544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 23:22:29.141363 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.141344 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.144227 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.144208 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 23:22:29.144894 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.144878 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.146572 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.146553 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 06 23:22:29.174757 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.174675 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 23:22:29.176103 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.176088 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-z62jv\"" Jul 06 23:22:29.176839 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.176823 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-cbcdr\"" Jul 06 23:22:29.195069 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.195050 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ctk6j\"" Jul 06 23:22:29.209475 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.209455 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:22:29.216282 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.216267 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 06 23:22:29.221303 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.221288 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.228194 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.228175 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 23:22:29.228261 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.228207 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.230116 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.230102 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 23:22:29.230638 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.230625 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 23:22:29.249521 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.249500 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.254252 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.254239 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 23:22:29.255326 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.255313 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.257159 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.257143 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 23:22:29.259107 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.259093 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 23:22:29.260722 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.260709 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bs2s8\"" Jul 06 23:22:29.267453 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.267438 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 23:22:29.272470 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.272452 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-6sq42\"" Jul 06 23:22:29.272546 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.272471 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ds5zr\"" Jul 06 23:22:29.274079 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.274064 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 23:22:29.275797 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.275781 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 23:22:29.277193 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.277180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 23:22:29.280529 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.280513 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.283224 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.283208 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 23:22:29.286062 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.286047 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 23:22:29.287565 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.287551 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:22:29.294528 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.294512 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.295711 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.295696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 23:22:29.295822 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.295724 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 23:22:29.301481 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.301464 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.305575 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.305560 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f7t4d\"" Jul 06 23:22:29.312790 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.312771 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.315705 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.315691 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 23:22:29.315769 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.315725 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wjzvb\"" Jul 06 23:22:29.316397 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.316381 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 23:22:29.317227 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.317213 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 23:22:29.326004 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.325988 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 23:22:29.342487 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.342465 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.342583 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.342540 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.347989 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.347970 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7xhr7\"" Jul 06 23:22:29.348764 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.348747 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 06 23:22:29.349200 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.349179 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 23:22:29.374812 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.374793 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 23:22:29.381458 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.381444 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 23:22:29.385984 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.385969 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 23:22:29.392923 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.392906 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 23:22:29.393558 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.393541 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 23:22:29.395204 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.395189 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-279gpg6osqfm0\"" Jul 06 23:22:29.396065 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.396048 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.398564 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.398547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 23:22:29.400813 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.400796 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 23:22:29.403284 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.403267 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-eujn6c29h5m9p\"" Jul 06 23:22:29.403755 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.403738 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:22:29.428420 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.428371 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 23:22:29.453429 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.453413 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-86knk\"" Jul 06 23:22:29.453593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.453580 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pc5sr\"" Jul 06 23:22:29.460810 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.460792 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.462557 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.462541 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 06 23:22:29.465545 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.465529 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 23:22:29.465641 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.465598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 06 23:22:29.465699 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.465641 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dx78r\"" Jul 06 23:22:29.468791 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.468776 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 23:22:29.468870 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.468780 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c4v9t\"" Jul 06 23:22:29.485376 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.485356 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 06 23:22:29.485376 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.485377 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 23:22:29.486272 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.486255 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 23:22:29.492057 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.492041 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 23:22:29.504043 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.504024 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.528309 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.528280 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 23:22:29.537633 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.537613 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.558191 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.558173 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jmlxb\"" Jul 06 23:22:29.578001 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.577972 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 23:22:29.598258 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.598227 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njcl6\"" Jul 06 23:22:29.616988 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.616960 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.637386 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.637358 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 23:22:29.657080 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.657050 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 23:22:29.677328 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.677297 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 23:22:29.697860 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.697779 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-57bqg\"" Jul 06 23:22:29.718139 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.718103 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.737725 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.737696 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 23:22:29.757520 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.757495 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 23:22:29.777790 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.777759 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 23:22:29.798525 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.798491 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.817425 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.817395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 06 23:22:29.838038 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.838007 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kjn6w\"" Jul 06 23:22:29.857384 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.857355 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 23:22:29.878025 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.877996 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 23:22:29.898553 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.898526 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 23:22:29.918133 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.918105 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-q5276\"" Jul 06 23:22:29.937653 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.937622 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 23:22:29.958470 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.958388 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 23:22:29.978061 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.978032 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 06 23:22:29.998203 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:29.998169 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 23:22:30.017695 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.017663 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"default-dockercfg-42l67\"" Jul 06 23:22:30.038112 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.038081 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 23:22:30.057328 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.057298 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 23:22:30.077400 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.077372 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-39ut8t85kt3o4\"" Jul 06 23:22:30.097647 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.097621 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 23:22:30.118030 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.118005 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 23:22:30.137631 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.137600 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"auth-test-model-2cc96a-kserve-self-signed-certs\"" Jul 06 23:22:30.158212 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.158188 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 23:22:30.177296 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.177275 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 23:22:30.197631 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.197607 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 23:22:30.217964 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.217888 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 23:22:30.237481 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.237450 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 23:22:30.257508 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:30.257483 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 23:22:44.387407 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.387311 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52"] Jul 06 23:22:44.391617 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.391595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.394998 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.394970 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"default-dockercfg-rhmgp\"" Jul 06 23:22:44.394998 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.394987 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"auth-test-model-ca352b-kserve-self-signed-certs\"" Jul 06 23:22:44.395212 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.395012 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"kube-root-ca.crt\"" Jul 06 23:22:44.395212 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.395012 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"openshift-service-ca.crt\"" Jul 06 23:22:44.396846 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.396824 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52"] Jul 06 23:22:44.420754 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.420724 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:22:44.475879 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.475840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-model-cache\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.475879 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.475880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-dshm\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.476154 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.475906 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-home\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.476154 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.475954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq6h4\" (UniqueName: \"kubernetes.io/projected/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-kube-api-access-tq6h4\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.476154 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.476015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tmp-dir\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.476154 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.476102 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tls-certs\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.576638 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.576603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tls-certs\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.576811 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.576664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-model-cache\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.576811 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.576682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-dshm\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.576811 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.576705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-home\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.576811 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.576720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq6h4\" (UniqueName: \"kubernetes.io/projected/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-kube-api-access-tq6h4\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.576811 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.576739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tmp-dir\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.577140 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.577101 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tmp-dir\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.577220 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.577204 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-model-cache\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.577380 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.577360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-home\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.579315 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.579290 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-dshm\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.579418 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.579378 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tls-certs\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.584782 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.584760 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq6h4\" (UniqueName: \"kubernetes.io/projected/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-kube-api-access-tq6h4\") pod \"auth-test-model-ca352b-kserve-b6df49459-58k52\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.702774 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.702738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:44.837077 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:44.837052 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52"] Jul 06 23:22:44.839090 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:22:44.839053 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20b61bd1_dc9b_4bba_932d_98c1ee53c18e.slice/crio-baca4fb56574284bb2ea49afb3b71231f152cafae2dd7d3172209d420dc4bc80 WatchSource:0}: Error finding container baca4fb56574284bb2ea49afb3b71231f152cafae2dd7d3172209d420dc4bc80: Status 404 returned error can't find the container with id baca4fb56574284bb2ea49afb3b71231f152cafae2dd7d3172209d420dc4bc80 Jul 06 23:22:45.783400 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:45.783367 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" event={"ID":"20b61bd1-dc9b-4bba-932d-98c1ee53c18e","Type":"ContainerStarted","Data":"d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447"} Jul 06 23:22:45.783800 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:45.783410 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" event={"ID":"20b61bd1-dc9b-4bba-932d-98c1ee53c18e","Type":"ContainerStarted","Data":"baca4fb56574284bb2ea49afb3b71231f152cafae2dd7d3172209d420dc4bc80"} Jul 06 23:22:45.801354 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:45.801301 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" podStartSLOduration=1.801283712 podStartE2EDuration="1.801283712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:22:45.800541867 +0000 UTC m=+2157.936266423" watchObservedRunningTime="2026-07-06 23:22:45.801283712 +0000 UTC m=+2157.937008267" Jul 06 23:22:47.980418 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:47.980374 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:22:54.703015 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:54.702980 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:54.703015 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:54.703019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:54.715263 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:54.715238 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:22:54.840266 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:22:54.840236 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:23:06.192978 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:23:06.192942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:23:19.779119 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:23:19.779082 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:23:36.883113 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:23:36.883074 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:23:51.778916 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:23:51.778879 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:24:01.616304 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:01.616232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:24:01.676940 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:01.676910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:24:09.380624 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:09.380586 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:24:22.118355 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:22.118279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:24:27.379987 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:27.379944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:24:40.966512 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:40.966471 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246"] Jul 06 23:24:40.985012 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:40.984983 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:24:41.860503 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:41.860468 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52"] Jul 06 23:24:41.876030 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:41.876004 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:24:44.970837 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:44.970806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:24:47.068179 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.068139 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j"] Jul 06 23:24:47.071744 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.071727 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.074451 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.074430 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"sub-test-model-2cc96a-kserve-self-signed-certs\"" Jul 06 23:24:47.082736 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.082715 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j"] Jul 06 23:24:47.085630 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.085602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-home\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.085740 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.085651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-dshm\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.085809 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.085728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tmp-dir\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.085809 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.085792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-model-cache\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.085945 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.085924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4695\" (UniqueName: \"kubernetes.io/projected/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-kube-api-access-w4695\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.085984 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.085971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tls-certs\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.095356 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.095328 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:24:47.186887 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.186856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tls-certs\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.187070 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.186919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-home\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.187070 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.186950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-dshm\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.187070 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.186979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tmp-dir\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.187070 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.187006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-model-cache\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.187779 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.187755 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-home\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.187963 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.187793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tmp-dir\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.188068 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.187159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4695\" (UniqueName: \"kubernetes.io/projected/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-kube-api-access-w4695\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.188433 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.187841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-model-cache\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.190451 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.190421 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-dshm\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.195181 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.194891 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tls-certs\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.196987 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.196968 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4695\" (UniqueName: \"kubernetes.io/projected/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-kube-api-access-w4695\") pod \"sub-test-model-2cc96a-kserve-657df9fb75-qgw5j\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.383486 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.383403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:47.518185 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:47.518157 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j"] Jul 06 23:24:47.519510 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:24:47.519484 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19e76056_a6a3_4ab4_8a4d_6b6a511088a1.slice/crio-dc6fc85b3a40a492554780826771690a4af012c75513e23e8dcaa4520e24ed07 WatchSource:0}: Error finding container dc6fc85b3a40a492554780826771690a4af012c75513e23e8dcaa4520e24ed07: Status 404 returned error can't find the container with id dc6fc85b3a40a492554780826771690a4af012c75513e23e8dcaa4520e24ed07 Jul 06 23:24:48.393194 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:48.393156 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" event={"ID":"19e76056-a6a3-4ab4-8a4d-6b6a511088a1","Type":"ContainerStarted","Data":"f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493"} Jul 06 23:24:48.393592 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:48.393206 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" event={"ID":"19e76056-a6a3-4ab4-8a4d-6b6a511088a1","Type":"ContainerStarted","Data":"dc6fc85b3a40a492554780826771690a4af012c75513e23e8dcaa4520e24ed07"} Jul 06 23:24:48.411790 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:48.411733 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" podStartSLOduration=1.411712961 podStartE2EDuration="1.411712961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:24:48.410021405 +0000 UTC m=+2280.545745969" watchObservedRunningTime="2026-07-06 23:24:48.411712961 +0000 UTC m=+2280.547437521" Jul 06 23:24:55.991741 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:55.991681 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" podUID="6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" containerName="main" containerID="cri-o://c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f" gracePeriod=45 Jul 06 23:24:56.246816 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.246753 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:24:56.272615 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272587 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tmp-dir\") pod \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " Jul 06 23:24:56.272777 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272650 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tls-certs\") pod \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " Jul 06 23:24:56.272777 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272677 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-home\") pod \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " Jul 06 23:24:56.272777 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272700 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-dshm\") pod \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " Jul 06 23:24:56.272777 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272733 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-model-cache\") pod \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " Jul 06 23:24:56.273004 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272781 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hr4g9\" (UniqueName: \"kubernetes.io/projected/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-kube-api-access-hr4g9\") pod \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\" (UID: \"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0\") " Jul 06 23:24:56.273004 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272847 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" (UID: "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:56.273004 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.272931 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-home" (OuterVolumeSpecName: "home") pod "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" (UID: "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:56.273183 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.273079 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" (UID: "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:56.273243 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.273187 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:56.273243 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.273201 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:56.273243 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.273234 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:56.274875 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.274851 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-dshm" (OuterVolumeSpecName: "dshm") pod "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" (UID: "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:56.274981 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.274910 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-kube-api-access-hr4g9" (OuterVolumeSpecName: "kube-api-access-hr4g9") pod "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" (UID: "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0"). InnerVolumeSpecName "kube-api-access-hr4g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:24:56.274981 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.274941 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" (UID: "6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:24:56.373736 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.373701 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:56.373736 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.373731 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr4g9\" (UniqueName: \"kubernetes.io/projected/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-kube-api-access-hr4g9\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:56.373736 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.373741 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:56.440265 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.440239 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" containerID="c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f" exitCode=0 Jul 06 23:24:56.440424 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.440271 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" event={"ID":"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0","Type":"ContainerDied","Data":"c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f"} Jul 06 23:24:56.440424 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.440282 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" Jul 06 23:24:56.440424 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.440294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246" event={"ID":"6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0","Type":"ContainerDied","Data":"681c57789e53cc37f6d6045c0914590fdd8c0e60bd22d2b65e4970e063d08732"} Jul 06 23:24:56.440424 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.440310 2579 scope.go:117] "RemoveContainer" containerID="c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f" Jul 06 23:24:56.448638 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.448619 2579 scope.go:117] "RemoveContainer" containerID="c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f" Jul 06 23:24:56.448882 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:24:56.448864 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f\": container with ID starting with c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f not found: ID does not exist" containerID="c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f" Jul 06 23:24:56.448949 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.448893 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f"} err="failed to get container status \"c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f\": rpc error: code = NotFound desc = could not find container \"c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f\": container with ID starting with c1bd3c9fc9798cabd55f8a2f4fbda3c0321954ff368cbb6558fc92cba30c999f not found: ID does not exist" Jul 06 23:24:56.457336 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.457312 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246"] Jul 06 23:24:56.459772 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.459750 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/auth-test-model-2cc96a-kserve-64b9fb659f-d5246"] Jul 06 23:24:56.881102 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:56.881026 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" podUID="20b61bd1-dc9b-4bba-932d-98c1ee53c18e" containerName="main" containerID="cri-o://d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447" gracePeriod=45 Jul 06 23:24:57.141653 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.141589 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:24:57.182378 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182341 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tls-certs\") pod \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " Jul 06 23:24:57.182570 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182417 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-dshm\") pod \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " Jul 06 23:24:57.182570 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182455 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-home\") pod \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " Jul 06 23:24:57.182570 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-model-cache\") pod \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " Jul 06 23:24:57.182570 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182542 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq6h4\" (UniqueName: \"kubernetes.io/projected/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-kube-api-access-tq6h4\") pod \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " Jul 06 23:24:57.182570 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182566 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tmp-dir\") pod \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\" (UID: \"20b61bd1-dc9b-4bba-932d-98c1ee53c18e\") " Jul 06 23:24:57.182835 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182732 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-model-cache" (OuterVolumeSpecName: "model-cache") pod "20b61bd1-dc9b-4bba-932d-98c1ee53c18e" (UID: "20b61bd1-dc9b-4bba-932d-98c1ee53c18e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:57.182835 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182757 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-home" (OuterVolumeSpecName: "home") pod "20b61bd1-dc9b-4bba-932d-98c1ee53c18e" (UID: "20b61bd1-dc9b-4bba-932d-98c1ee53c18e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:57.182913 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182888 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20b61bd1-dc9b-4bba-932d-98c1ee53c18e" (UID: "20b61bd1-dc9b-4bba-932d-98c1ee53c18e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:57.182989 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182965 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:57.182989 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182984 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:57.183092 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.182992 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:57.184654 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.184628 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20b61bd1-dc9b-4bba-932d-98c1ee53c18e" (UID: "20b61bd1-dc9b-4bba-932d-98c1ee53c18e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:24:57.184764 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.184688 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-kube-api-access-tq6h4" (OuterVolumeSpecName: "kube-api-access-tq6h4") pod "20b61bd1-dc9b-4bba-932d-98c1ee53c18e" (UID: "20b61bd1-dc9b-4bba-932d-98c1ee53c18e"). InnerVolumeSpecName "kube-api-access-tq6h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:24:57.184923 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.184902 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-dshm" (OuterVolumeSpecName: "dshm") pod "20b61bd1-dc9b-4bba-932d-98c1ee53c18e" (UID: "20b61bd1-dc9b-4bba-932d-98c1ee53c18e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:24:57.284344 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.284296 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:57.284344 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.284335 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:57.284344 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.284346 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq6h4\" (UniqueName: \"kubernetes.io/projected/20b61bd1-dc9b-4bba-932d-98c1ee53c18e-kube-api-access-tq6h4\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:24:57.383908 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.383873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:57.384099 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.383927 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:57.396110 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.396087 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:57.453717 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.453686 2579 generic.go:356] "Generic (PLEG): container finished" podID="20b61bd1-dc9b-4bba-932d-98c1ee53c18e" containerID="d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447" exitCode=0 Jul 06 23:24:57.453878 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.453731 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" Jul 06 23:24:57.453878 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.453765 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" event={"ID":"20b61bd1-dc9b-4bba-932d-98c1ee53c18e","Type":"ContainerDied","Data":"d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447"} Jul 06 23:24:57.453878 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.453802 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52" event={"ID":"20b61bd1-dc9b-4bba-932d-98c1ee53c18e","Type":"ContainerDied","Data":"baca4fb56574284bb2ea49afb3b71231f152cafae2dd7d3172209d420dc4bc80"} Jul 06 23:24:57.453878 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.453823 2579 scope.go:117] "RemoveContainer" containerID="d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447" Jul 06 23:24:57.464381 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.464344 2579 scope.go:117] "RemoveContainer" containerID="d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447" Jul 06 23:24:57.464633 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:24:57.464590 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447\": container with ID starting with d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447 not found: ID does not exist" containerID="d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447" Jul 06 23:24:57.464683 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.464645 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447"} err="failed to get container status \"d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447\": rpc error: code = NotFound desc = could not find container \"d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447\": container with ID starting with d671ab23a1fa5b1ce454f1a773a0aa6c9369c9b669f6c367fec53c15c80db447 not found: ID does not exist" Jul 06 23:24:57.467534 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.467517 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:24:57.475420 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.475400 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52"] Jul 06 23:24:57.479110 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:57.479092 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/auth-test-model-ca352b-kserve-b6df49459-58k52"] Jul 06 23:24:58.427404 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:58.427377 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20b61bd1-dc9b-4bba-932d-98c1ee53c18e" path="/var/lib/kubelet/pods/20b61bd1-dc9b-4bba-932d-98c1ee53c18e/volumes" Jul 06 23:24:58.427748 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:24:58.427735 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" path="/var/lib/kubelet/pods/6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0/volumes" Jul 06 23:25:00.179942 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:00.179903 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:25:15.183312 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.183276 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx"] Jul 06 23:25:15.183674 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.183649 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b61bd1-dc9b-4bba-932d-98c1ee53c18e" containerName="main" Jul 06 23:25:15.183674 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.183659 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b61bd1-dc9b-4bba-932d-98c1ee53c18e" containerName="main" Jul 06 23:25:15.183746 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.183683 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" containerName="main" Jul 06 23:25:15.183746 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.183690 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" containerName="main" Jul 06 23:25:15.183808 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.183749 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="20b61bd1-dc9b-4bba-932d-98c1ee53c18e" containerName="main" Jul 06 23:25:15.183808 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.183757 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b028e4b-87e7-4f5d-a8fb-6d1ee76725b0" containerName="main" Jul 06 23:25:15.188597 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.188577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.193781 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.191888 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"kube-root-ca.crt\"" Jul 06 23:25:15.196628 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.196603 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"openshift-service-ca.crt\"" Jul 06 23:25:15.197782 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.196914 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"sub-test-model-ca352b-kserve-self-signed-certs\"" Jul 06 23:25:15.197782 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.197051 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"default-dockercfg-rhmgp\"" Jul 06 23:25:15.199836 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.199806 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx"] Jul 06 23:25:15.200833 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.200809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:25:15.354563 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.354517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-tmp-dir\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.354563 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.354578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-home\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.354799 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.354602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-dshm\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.354799 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.354620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55pr9\" (UniqueName: \"kubernetes.io/projected/0e91c834-7bd0-4401-b5be-04f7e06253c3-kube-api-access-55pr9\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.354799 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.354640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-model-cache\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.354799 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.354653 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e91c834-7bd0-4401-b5be-04f7e06253c3-tls-certs\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456066 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-home\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456066 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-dshm\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456066 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55pr9\" (UniqueName: \"kubernetes.io/projected/0e91c834-7bd0-4401-b5be-04f7e06253c3-kube-api-access-55pr9\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456066 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-model-cache\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456472 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456080 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e91c834-7bd0-4401-b5be-04f7e06253c3-tls-certs\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456472 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-tmp-dir\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456472 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-home\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456624 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456485 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-model-cache\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.456663 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.456629 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-tmp-dir\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.458364 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.458343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-dshm\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.458720 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.458701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e91c834-7bd0-4401-b5be-04f7e06253c3-tls-certs\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.463444 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.463423 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55pr9\" (UniqueName: \"kubernetes.io/projected/0e91c834-7bd0-4401-b5be-04f7e06253c3-kube-api-access-55pr9\") pod \"sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.501251 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.501224 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:15.636573 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:15.636540 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx"] Jul 06 23:25:15.637703 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:25:15.637676 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e91c834_7bd0_4401_b5be_04f7e06253c3.slice/crio-19253dc74c4300bdbac5778cd81b9ad4381bf6d350a4187b7bd8b987fcc949b8 WatchSource:0}: Error finding container 19253dc74c4300bdbac5778cd81b9ad4381bf6d350a4187b7bd8b987fcc949b8: Status 404 returned error can't find the container with id 19253dc74c4300bdbac5778cd81b9ad4381bf6d350a4187b7bd8b987fcc949b8 Jul 06 23:25:16.546021 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:16.545985 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" event={"ID":"0e91c834-7bd0-4401-b5be-04f7e06253c3","Type":"ContainerStarted","Data":"25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f"} Jul 06 23:25:16.546406 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:16.546032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" event={"ID":"0e91c834-7bd0-4401-b5be-04f7e06253c3","Type":"ContainerStarted","Data":"19253dc74c4300bdbac5778cd81b9ad4381bf6d350a4187b7bd8b987fcc949b8"} Jul 06 23:25:16.562700 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:16.562656 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" podStartSLOduration=1.562642485 podStartE2EDuration="1.562642485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:25:16.560502193 +0000 UTC m=+2308.696226752" watchObservedRunningTime="2026-07-06 23:25:16.562642485 +0000 UTC m=+2308.698367037" Jul 06 23:25:19.482866 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:19.482829 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:25:25.502043 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:25.502003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:25.502488 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:25.502056 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:25.514395 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:25.514369 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:25.600652 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:25.600630 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:25:37.077707 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:37.077672 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:25:52.020477 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:52.020441 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j"] Jul 06 23:25:52.039167 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:52.039140 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:25:52.885914 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:52.885884 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx"] Jul 06 23:25:52.898858 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:52.898831 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:25:55.980409 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:25:55.980370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:26:00.181288 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:00.181254 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:00.279322 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:00.279291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:07.040481 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.040441 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" podUID="19e76056-a6a3-4ab4-8a4d-6b6a511088a1" containerName="main" containerID="cri-o://f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493" gracePeriod=45 Jul 06 23:26:07.283412 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.283382 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:26:07.344984 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.344916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4695\" (UniqueName: \"kubernetes.io/projected/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-kube-api-access-w4695\") pod \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " Jul 06 23:26:07.344984 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.344961 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tmp-dir\") pod \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " Jul 06 23:26:07.345165 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345004 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-model-cache\") pod \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " Jul 06 23:26:07.345165 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345084 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tls-certs\") pod \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " Jul 06 23:26:07.345165 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345160 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-dshm\") pod \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " Jul 06 23:26:07.345301 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345185 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-home\") pod \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\" (UID: \"19e76056-a6a3-4ab4-8a4d-6b6a511088a1\") " Jul 06 23:26:07.345363 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345296 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19e76056-a6a3-4ab4-8a4d-6b6a511088a1" (UID: "19e76056-a6a3-4ab4-8a4d-6b6a511088a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:07.345363 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345325 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-model-cache" (OuterVolumeSpecName: "model-cache") pod "19e76056-a6a3-4ab4-8a4d-6b6a511088a1" (UID: "19e76056-a6a3-4ab4-8a4d-6b6a511088a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:07.345537 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345508 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-home" (OuterVolumeSpecName: "home") pod "19e76056-a6a3-4ab4-8a4d-6b6a511088a1" (UID: "19e76056-a6a3-4ab4-8a4d-6b6a511088a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:07.345626 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345569 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:07.345626 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.345582 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:07.347006 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.346982 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-kube-api-access-w4695" (OuterVolumeSpecName: "kube-api-access-w4695") pod "19e76056-a6a3-4ab4-8a4d-6b6a511088a1" (UID: "19e76056-a6a3-4ab4-8a4d-6b6a511088a1"). InnerVolumeSpecName "kube-api-access-w4695". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:26:07.347441 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.347418 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19e76056-a6a3-4ab4-8a4d-6b6a511088a1" (UID: "19e76056-a6a3-4ab4-8a4d-6b6a511088a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:26:07.347531 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.347435 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-dshm" (OuterVolumeSpecName: "dshm") pod "19e76056-a6a3-4ab4-8a4d-6b6a511088a1" (UID: "19e76056-a6a3-4ab4-8a4d-6b6a511088a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:07.446505 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.446481 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:07.446505 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.446503 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:07.446642 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.446511 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:07.446642 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.446519 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4695\" (UniqueName: \"kubernetes.io/projected/19e76056-a6a3-4ab4-8a4d-6b6a511088a1-kube-api-access-w4695\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:07.792516 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.792482 2579 generic.go:356] "Generic (PLEG): container finished" podID="19e76056-a6a3-4ab4-8a4d-6b6a511088a1" containerID="f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493" exitCode=0 Jul 06 23:26:07.792685 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.792534 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" Jul 06 23:26:07.792685 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.792561 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" event={"ID":"19e76056-a6a3-4ab4-8a4d-6b6a511088a1","Type":"ContainerDied","Data":"f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493"} Jul 06 23:26:07.792685 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.792606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j" event={"ID":"19e76056-a6a3-4ab4-8a4d-6b6a511088a1","Type":"ContainerDied","Data":"dc6fc85b3a40a492554780826771690a4af012c75513e23e8dcaa4520e24ed07"} Jul 06 23:26:07.792685 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.792632 2579 scope.go:117] "RemoveContainer" containerID="f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493" Jul 06 23:26:07.801420 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.801402 2579 scope.go:117] "RemoveContainer" containerID="f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493" Jul 06 23:26:07.801663 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:26:07.801636 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493\": container with ID starting with f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493 not found: ID does not exist" containerID="f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493" Jul 06 23:26:07.801706 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.801662 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493"} err="failed to get container status \"f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493\": rpc error: code = NotFound desc = could not find container \"f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493\": container with ID starting with f09c2a61b63b5a33afc550e8dd85faaf6eaa44c63baa1ea495dd595648845493 not found: ID does not exist" Jul 06 23:26:07.816247 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.816218 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j"] Jul 06 23:26:07.818551 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.818534 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/sub-test-model-2cc96a-kserve-657df9fb75-qgw5j"] Jul 06 23:26:07.905845 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:07.905808 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" podUID="0e91c834-7bd0-4401-b5be-04f7e06253c3" containerName="main" containerID="cri-o://25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f" gracePeriod=45 Jul 06 23:26:08.153117 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.153091 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:26:08.254064 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254033 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55pr9\" (UniqueName: \"kubernetes.io/projected/0e91c834-7bd0-4401-b5be-04f7e06253c3-kube-api-access-55pr9\") pod \"0e91c834-7bd0-4401-b5be-04f7e06253c3\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " Jul 06 23:26:08.254064 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254070 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e91c834-7bd0-4401-b5be-04f7e06253c3-tls-certs\") pod \"0e91c834-7bd0-4401-b5be-04f7e06253c3\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " Jul 06 23:26:08.254381 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254106 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-home\") pod \"0e91c834-7bd0-4401-b5be-04f7e06253c3\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " Jul 06 23:26:08.254381 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254185 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-model-cache\") pod \"0e91c834-7bd0-4401-b5be-04f7e06253c3\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " Jul 06 23:26:08.254381 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254249 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-tmp-dir\") pod \"0e91c834-7bd0-4401-b5be-04f7e06253c3\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " Jul 06 23:26:08.254381 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254300 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-dshm\") pod \"0e91c834-7bd0-4401-b5be-04f7e06253c3\" (UID: \"0e91c834-7bd0-4401-b5be-04f7e06253c3\") " Jul 06 23:26:08.254575 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254437 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-home" (OuterVolumeSpecName: "home") pod "0e91c834-7bd0-4401-b5be-04f7e06253c3" (UID: "0e91c834-7bd0-4401-b5be-04f7e06253c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:08.254575 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254471 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e91c834-7bd0-4401-b5be-04f7e06253c3" (UID: "0e91c834-7bd0-4401-b5be-04f7e06253c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:08.254575 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254511 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e91c834-7bd0-4401-b5be-04f7e06253c3" (UID: "0e91c834-7bd0-4401-b5be-04f7e06253c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:08.254718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254637 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:08.254718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254656 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:08.254718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.254667 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:08.256159 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.256118 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e91c834-7bd0-4401-b5be-04f7e06253c3-kube-api-access-55pr9" (OuterVolumeSpecName: "kube-api-access-55pr9") pod "0e91c834-7bd0-4401-b5be-04f7e06253c3" (UID: "0e91c834-7bd0-4401-b5be-04f7e06253c3"). InnerVolumeSpecName "kube-api-access-55pr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:26:08.256284 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.256265 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e91c834-7bd0-4401-b5be-04f7e06253c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e91c834-7bd0-4401-b5be-04f7e06253c3" (UID: "0e91c834-7bd0-4401-b5be-04f7e06253c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:26:08.256372 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.256351 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-dshm" (OuterVolumeSpecName: "dshm") pod "0e91c834-7bd0-4401-b5be-04f7e06253c3" (UID: "0e91c834-7bd0-4401-b5be-04f7e06253c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:26:08.356194 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.356086 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e91c834-7bd0-4401-b5be-04f7e06253c3-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:08.356194 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.356112 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55pr9\" (UniqueName: \"kubernetes.io/projected/0e91c834-7bd0-4401-b5be-04f7e06253c3-kube-api-access-55pr9\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:08.356194 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.356137 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e91c834-7bd0-4401-b5be-04f7e06253c3-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:26:08.428479 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.428449 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e76056-a6a3-4ab4-8a4d-6b6a511088a1" path="/var/lib/kubelet/pods/19e76056-a6a3-4ab4-8a4d-6b6a511088a1/volumes" Jul 06 23:26:08.799289 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.799257 2579 generic.go:356] "Generic (PLEG): container finished" podID="0e91c834-7bd0-4401-b5be-04f7e06253c3" containerID="25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f" exitCode=0 Jul 06 23:26:08.799482 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.799302 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" Jul 06 23:26:08.799482 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.799307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" event={"ID":"0e91c834-7bd0-4401-b5be-04f7e06253c3","Type":"ContainerDied","Data":"25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f"} Jul 06 23:26:08.799482 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.799342 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx" event={"ID":"0e91c834-7bd0-4401-b5be-04f7e06253c3","Type":"ContainerDied","Data":"19253dc74c4300bdbac5778cd81b9ad4381bf6d350a4187b7bd8b987fcc949b8"} Jul 06 23:26:08.799482 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.799363 2579 scope.go:117] "RemoveContainer" containerID="25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f" Jul 06 23:26:08.809858 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.809836 2579 scope.go:117] "RemoveContainer" containerID="25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f" Jul 06 23:26:08.810366 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:26:08.810345 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f\": container with ID starting with 25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f not found: ID does not exist" containerID="25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f" Jul 06 23:26:08.810463 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.810384 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f"} err="failed to get container status \"25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f\": rpc error: code = NotFound desc = could not find container \"25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f\": container with ID starting with 25fa51fa4088a9aa21939b82c9b0ce83fa922b2da13898f24ccd8cab0651768f not found: ID does not exist" Jul 06 23:26:08.815955 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.815933 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx"] Jul 06 23:26:08.819911 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:08.819886 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/sub-test-model-ca352b-kserve-bcbf44dc9-ddvcx"] Jul 06 23:26:10.429407 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:10.429371 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e91c834-7bd0-4401-b5be-04f7e06253c3" path="/var/lib/kubelet/pods/0e91c834-7bd0-4401-b5be-04f7e06253c3/volumes" Jul 06 23:26:17.781909 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.781870 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw"] Jul 06 23:26:17.782490 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.782474 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e76056-a6a3-4ab4-8a4d-6b6a511088a1" containerName="main" Jul 06 23:26:17.782533 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.782493 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e76056-a6a3-4ab4-8a4d-6b6a511088a1" containerName="main" Jul 06 23:26:17.782533 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.782512 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e91c834-7bd0-4401-b5be-04f7e06253c3" containerName="main" Jul 06 23:26:17.782533 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.782520 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e91c834-7bd0-4401-b5be-04f7e06253c3" containerName="main" Jul 06 23:26:17.782643 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.782632 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e91c834-7bd0-4401-b5be-04f7e06253c3" containerName="main" Jul 06 23:26:17.782677 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.782648 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e76056-a6a3-4ab4-8a4d-6b6a511088a1" containerName="main" Jul 06 23:26:17.787267 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.787248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.791205 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.791180 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"kube-root-ca.crt\"" Jul 06 23:26:17.791534 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.791466 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"rate-test-model-2cc96a-kserve-self-signed-certs\"" Jul 06 23:26:17.791534 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.791485 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"openshift-service-ca.crt\"" Jul 06 23:26:17.791534 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.791519 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cc96a\"/\"default-dockercfg-42l67\"" Jul 06 23:26:17.795371 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.795348 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw"] Jul 06 23:26:17.809769 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.809749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:17.835817 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.835789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-dshm\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.835909 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.835836 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tmp-dir\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.835909 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.835898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-home\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.835981 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.835927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-model-cache\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.835981 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.835946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzl9\" (UniqueName: \"kubernetes.io/projected/93e1de61-9e8e-4b42-9ddc-d8390e79f696-kube-api-access-htzl9\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.835981 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.835967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tls-certs\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937147 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-home\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937338 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-model-cache\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937338 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzl9\" (UniqueName: \"kubernetes.io/projected/93e1de61-9e8e-4b42-9ddc-d8390e79f696-kube-api-access-htzl9\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937338 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tls-certs\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937338 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-dshm\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937338 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tmp-dir\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937605 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937532 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-home\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937605 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937567 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-model-cache\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.937605 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.937584 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tmp-dir\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.939396 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.939377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-dshm\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.939781 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.939762 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tls-certs\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:17.945072 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:17.945049 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzl9\" (UniqueName: \"kubernetes.io/projected/93e1de61-9e8e-4b42-9ddc-d8390e79f696-kube-api-access-htzl9\") pod \"rate-test-model-2cc96a-kserve-8588f9d67-xqxkw\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:18.097803 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:18.097728 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:18.224220 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:18.224188 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw"] Jul 06 23:26:18.225277 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:26:18.225253 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93e1de61_9e8e_4b42_9ddc_d8390e79f696.slice/crio-6b23bb77cd9699f02ac15367f33f5fbe5cfc1f575577afd12d1c868062390422 WatchSource:0}: Error finding container 6b23bb77cd9699f02ac15367f33f5fbe5cfc1f575577afd12d1c868062390422: Status 404 returned error can't find the container with id 6b23bb77cd9699f02ac15367f33f5fbe5cfc1f575577afd12d1c868062390422 Jul 06 23:26:18.848084 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:18.848042 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" event={"ID":"93e1de61-9e8e-4b42-9ddc-d8390e79f696","Type":"ContainerStarted","Data":"ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802"} Jul 06 23:26:18.848084 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:18.848084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" event={"ID":"93e1de61-9e8e-4b42-9ddc-d8390e79f696","Type":"ContainerStarted","Data":"6b23bb77cd9699f02ac15367f33f5fbe5cfc1f575577afd12d1c868062390422"} Jul 06 23:26:18.864818 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:18.864770 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" podStartSLOduration=1.864755745 podStartE2EDuration="1.864755745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:26:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:18.863350503 +0000 UTC m=+2370.999075081" watchObservedRunningTime="2026-07-06 23:26:18.864755745 +0000 UTC m=+2371.000480299" Jul 06 23:26:21.480994 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:21.480956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:26:28.098279 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:28.098239 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:28.098736 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:28.098292 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:28.109995 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:28.109967 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:28.915025 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:28.914996 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:26:37.278049 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.278012 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb"] Jul 06 23:26:37.281589 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.281573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.284034 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.284010 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"kube-root-ca.crt\"" Jul 06 23:26:37.284172 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.284040 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"openshift-service-ca.crt\"" Jul 06 23:26:37.285187 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.285158 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"rate-test-model-ca352b-kserve-self-signed-certs\"" Jul 06 23:26:37.285187 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.285174 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ca352b\"/\"default-dockercfg-rhmgp\"" Jul 06 23:26:37.293557 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.293534 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb"] Jul 06 23:26:37.305232 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.305209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:37.429336 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.429307 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-home\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.429537 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.429343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-tmp-dir\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.429537 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.429361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-dshm\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.429537 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.429459 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0530bb-f91c-4b27-b552-554cd6002232-tls-certs\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.429537 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.429512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-model-cache\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.429693 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.429559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg8z\" (UniqueName: \"kubernetes.io/projected/5a0530bb-f91c-4b27-b552-554cd6002232-kube-api-access-shg8z\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530223 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-home\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530223 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-tmp-dir\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530223 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-dshm\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530473 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0530bb-f91c-4b27-b552-554cd6002232-tls-certs\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530473 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-model-cache\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530473 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg8z\" (UniqueName: \"kubernetes.io/projected/5a0530bb-f91c-4b27-b552-554cd6002232-kube-api-access-shg8z\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530597 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530521 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-home\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530597 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530539 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-tmp-dir\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.530719 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.530703 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-model-cache\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.532575 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.532554 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-dshm\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.532860 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.532841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0530bb-f91c-4b27-b552-554cd6002232-tls-certs\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.538734 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.538714 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg8z\" (UniqueName: \"kubernetes.io/projected/5a0530bb-f91c-4b27-b552-554cd6002232-kube-api-access-shg8z\") pod \"rate-test-model-ca352b-kserve-6ddcbc886c-98prb\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.593656 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.593625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:37.725321 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.725295 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb"] Jul 06 23:26:37.725888 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:26:37.725868 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a0530bb_f91c_4b27_b552_554cd6002232.slice/crio-1aef4e17e6b688c4d3ff62a71143269df38b7f28687c94874360255e8c581f8d WatchSource:0}: Error finding container 1aef4e17e6b688c4d3ff62a71143269df38b7f28687c94874360255e8c581f8d: Status 404 returned error can't find the container with id 1aef4e17e6b688c4d3ff62a71143269df38b7f28687c94874360255e8c581f8d Jul 06 23:26:37.949063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.949027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" event={"ID":"5a0530bb-f91c-4b27-b552-554cd6002232","Type":"ContainerStarted","Data":"493bf786031e11688e4e2f61f788d7af578bc0273d9fb78d1c0043c061a9f8ae"} Jul 06 23:26:37.949063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.949070 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" event={"ID":"5a0530bb-f91c-4b27-b552-554cd6002232","Type":"ContainerStarted","Data":"1aef4e17e6b688c4d3ff62a71143269df38b7f28687c94874360255e8c581f8d"} Jul 06 23:26:37.965579 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:37.965531 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" podStartSLOduration=0.965514462 podStartE2EDuration="965.514462ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:26:37.964027597 +0000 UTC m=+2390.099752171" watchObservedRunningTime="2026-07-06 23:26:37.965514462 +0000 UTC m=+2390.101239016" Jul 06 23:26:39.422770 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:39.422737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:39.521628 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:39.521587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:41.476973 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:41.476929 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:26:47.593899 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:47.593862 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:47.594400 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:47.593916 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:47.605779 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:47.605758 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:48.006988 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:48.006961 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:26:48.378279 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:48.378158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:48.479545 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:48.479507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:26:48.496971 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:48.496940 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:26:48.504523 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:48.504503 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:26:59.274586 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:26:59.274548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:27:13.975907 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:13.975831 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:27:21.175509 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:21.175472 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw"] Jul 06 23:27:21.197846 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:21.197819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:21.988365 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:21.988332 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb"] Jul 06 23:27:22.005072 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:22.005026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:22.992519 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:22.992460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:23.377997 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:23.377890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:23.477906 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:23.477873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:23.578218 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:23.578178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:23.678601 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:23.678576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:26.779712 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:26.779683 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:27:28.456672 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:28.456645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:36.195088 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.195051 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" podUID="93e1de61-9e8e-4b42-9ddc-d8390e79f696" containerName="main" containerID="cri-o://ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802" gracePeriod=45 Jul 06 23:27:36.444643 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.444620 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:27:36.461139 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461063 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htzl9\" (UniqueName: \"kubernetes.io/projected/93e1de61-9e8e-4b42-9ddc-d8390e79f696-kube-api-access-htzl9\") pod \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " Jul 06 23:27:36.462071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461489 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tmp-dir\") pod \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " Jul 06 23:27:36.462071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461520 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-home\") pod \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " Jul 06 23:27:36.462071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461658 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-model-cache\") pod \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " Jul 06 23:27:36.462071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461737 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-dshm\") pod \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " Jul 06 23:27:36.462071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461752 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93e1de61-9e8e-4b42-9ddc-d8390e79f696" (UID: "93e1de61-9e8e-4b42-9ddc-d8390e79f696"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:36.462071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461782 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tls-certs\") pod \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\" (UID: \"93e1de61-9e8e-4b42-9ddc-d8390e79f696\") " Jul 06 23:27:36.462071 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.461910 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-home" (OuterVolumeSpecName: "home") pod "93e1de61-9e8e-4b42-9ddc-d8390e79f696" (UID: "93e1de61-9e8e-4b42-9ddc-d8390e79f696"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:36.462489 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.462193 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-model-cache" (OuterVolumeSpecName: "model-cache") pod "93e1de61-9e8e-4b42-9ddc-d8390e79f696" (UID: "93e1de61-9e8e-4b42-9ddc-d8390e79f696"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:36.462489 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.462201 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:36.462489 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.462342 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:36.464056 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.464033 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-dshm" (OuterVolumeSpecName: "dshm") pod "93e1de61-9e8e-4b42-9ddc-d8390e79f696" (UID: "93e1de61-9e8e-4b42-9ddc-d8390e79f696"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:36.464308 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.464276 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e1de61-9e8e-4b42-9ddc-d8390e79f696-kube-api-access-htzl9" (OuterVolumeSpecName: "kube-api-access-htzl9") pod "93e1de61-9e8e-4b42-9ddc-d8390e79f696" (UID: "93e1de61-9e8e-4b42-9ddc-d8390e79f696"). InnerVolumeSpecName "kube-api-access-htzl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:27:36.464406 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.464375 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93e1de61-9e8e-4b42-9ddc-d8390e79f696" (UID: "93e1de61-9e8e-4b42-9ddc-d8390e79f696"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:27:36.563379 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.563356 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:36.563379 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.563375 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93e1de61-9e8e-4b42-9ddc-d8390e79f696-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:36.563379 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.563384 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93e1de61-9e8e-4b42-9ddc-d8390e79f696-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:36.563579 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:36.563393 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzl9\" (UniqueName: \"kubernetes.io/projected/93e1de61-9e8e-4b42-9ddc-d8390e79f696-kube-api-access-htzl9\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:37.008011 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.007959 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" podUID="5a0530bb-f91c-4b27-b552-554cd6002232" containerName="main" containerID="cri-o://493bf786031e11688e4e2f61f788d7af578bc0273d9fb78d1c0043c061a9f8ae" gracePeriod=45 Jul 06 23:27:37.234861 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.234830 2579 generic.go:356] "Generic (PLEG): container finished" podID="93e1de61-9e8e-4b42-9ddc-d8390e79f696" containerID="ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802" exitCode=0 Jul 06 23:27:37.235265 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.234884 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" Jul 06 23:27:37.235265 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.234911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" event={"ID":"93e1de61-9e8e-4b42-9ddc-d8390e79f696","Type":"ContainerDied","Data":"ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802"} Jul 06 23:27:37.235265 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.234954 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw" event={"ID":"93e1de61-9e8e-4b42-9ddc-d8390e79f696","Type":"ContainerDied","Data":"6b23bb77cd9699f02ac15367f33f5fbe5cfc1f575577afd12d1c868062390422"} Jul 06 23:27:37.235265 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.234976 2579 scope.go:117] "RemoveContainer" containerID="ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802" Jul 06 23:27:37.236664 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.236637 2579 generic.go:356] "Generic (PLEG): container finished" podID="5a0530bb-f91c-4b27-b552-554cd6002232" containerID="493bf786031e11688e4e2f61f788d7af578bc0273d9fb78d1c0043c061a9f8ae" exitCode=0 Jul 06 23:27:37.236664 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.236664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" event={"ID":"5a0530bb-f91c-4b27-b552-554cd6002232","Type":"ContainerDied","Data":"493bf786031e11688e4e2f61f788d7af578bc0273d9fb78d1c0043c061a9f8ae"} Jul 06 23:27:37.256943 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.256926 2579 scope.go:117] "RemoveContainer" containerID="ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802" Jul 06 23:27:37.257321 ip-10-0-134-52 kubenswrapper[2579]: E0706 23:27:37.257295 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802\": container with ID starting with ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802 not found: ID does not exist" containerID="ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802" Jul 06 23:27:37.257413 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.257335 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802"} err="failed to get container status \"ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802\": rpc error: code = NotFound desc = could not find container \"ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802\": container with ID starting with ef90623252fca5b2faacee85bd75aa71a841dc677b3493127d8ee401a171c802 not found: ID does not exist" Jul 06 23:27:37.260821 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.260780 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:27:37.271062 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.271042 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw"] Jul 06 23:27:37.274893 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.274873 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2cc96a/rate-test-model-2cc96a-kserve-8588f9d67-xqxkw"] Jul 06 23:27:37.370405 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370369 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-dshm\") pod \"5a0530bb-f91c-4b27-b552-554cd6002232\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " Jul 06 23:27:37.370593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370415 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shg8z\" (UniqueName: \"kubernetes.io/projected/5a0530bb-f91c-4b27-b552-554cd6002232-kube-api-access-shg8z\") pod \"5a0530bb-f91c-4b27-b552-554cd6002232\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " Jul 06 23:27:37.370593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370443 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-tmp-dir\") pod \"5a0530bb-f91c-4b27-b552-554cd6002232\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " Jul 06 23:27:37.370593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370462 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0530bb-f91c-4b27-b552-554cd6002232-tls-certs\") pod \"5a0530bb-f91c-4b27-b552-554cd6002232\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " Jul 06 23:27:37.370593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370541 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-home\") pod \"5a0530bb-f91c-4b27-b552-554cd6002232\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " Jul 06 23:27:37.370593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370582 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-model-cache\") pod \"5a0530bb-f91c-4b27-b552-554cd6002232\" (UID: \"5a0530bb-f91c-4b27-b552-554cd6002232\") " Jul 06 23:27:37.370832 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370738 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a0530bb-f91c-4b27-b552-554cd6002232" (UID: "5a0530bb-f91c-4b27-b552-554cd6002232"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:37.370880 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370822 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-home" (OuterVolumeSpecName: "home") pod "5a0530bb-f91c-4b27-b552-554cd6002232" (UID: "5a0530bb-f91c-4b27-b552-554cd6002232"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:37.370880 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370833 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a0530bb-f91c-4b27-b552-554cd6002232" (UID: "5a0530bb-f91c-4b27-b552-554cd6002232"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:37.370956 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370942 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-tmp-dir\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:37.370992 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370960 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-home\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:37.370992 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.370971 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-model-cache\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:37.372603 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.372573 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0530bb-f91c-4b27-b552-554cd6002232-kube-api-access-shg8z" (OuterVolumeSpecName: "kube-api-access-shg8z") pod "5a0530bb-f91c-4b27-b552-554cd6002232" (UID: "5a0530bb-f91c-4b27-b552-554cd6002232"). InnerVolumeSpecName "kube-api-access-shg8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:27:37.372722 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.372626 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0530bb-f91c-4b27-b552-554cd6002232-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a0530bb-f91c-4b27-b552-554cd6002232" (UID: "5a0530bb-f91c-4b27-b552-554cd6002232"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:27:37.372722 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.372632 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-dshm" (OuterVolumeSpecName: "dshm") pod "5a0530bb-f91c-4b27-b552-554cd6002232" (UID: "5a0530bb-f91c-4b27-b552-554cd6002232"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:27:37.471938 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.471905 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a0530bb-f91c-4b27-b552-554cd6002232-dshm\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:37.471938 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.471937 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg8z\" (UniqueName: \"kubernetes.io/projected/5a0530bb-f91c-4b27-b552-554cd6002232-kube-api-access-shg8z\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:37.471938 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:37.471947 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a0530bb-f91c-4b27-b552-554cd6002232-tls-certs\") on node \"ip-10-0-134-52.ec2.internal\" DevicePath \"\"" Jul 06 23:27:38.243002 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:38.242972 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" Jul 06 23:27:38.243444 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:38.242980 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb" event={"ID":"5a0530bb-f91c-4b27-b552-554cd6002232","Type":"ContainerDied","Data":"1aef4e17e6b688c4d3ff62a71143269df38b7f28687c94874360255e8c581f8d"} Jul 06 23:27:38.243444 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:38.243096 2579 scope.go:117] "RemoveContainer" containerID="493bf786031e11688e4e2f61f788d7af578bc0273d9fb78d1c0043c061a9f8ae" Jul 06 23:27:38.265048 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:38.265025 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb"] Jul 06 23:27:38.268573 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:38.268553 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ca352b/rate-test-model-ca352b-kserve-6ddcbc886c-98prb"] Jul 06 23:27:38.427914 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:38.427883 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a0530bb-f91c-4b27-b552-554cd6002232" path="/var/lib/kubelet/pods/5a0530bb-f91c-4b27-b552-554cd6002232/volumes" Jul 06 23:27:38.428313 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:38.428299 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93e1de61-9e8e-4b42-9ddc-d8390e79f696" path="/var/lib/kubelet/pods/93e1de61-9e8e-4b42-9ddc-d8390e79f696/volumes" Jul 06 23:27:42.079674 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:42.079632 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:42.349915 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:42.349827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:42.712182 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:42.712151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:43.112485 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:43.112401 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:43.218844 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:43.218813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:43.362021 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:43.361985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:46.389718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:46.389682 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-m4d6x"] Jul 06 23:27:48.784945 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:48.784913 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:48.839566 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:48.839500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:49.033069 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:49.033035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:27:54.124081 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:54.124056 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gwnw8_551ce188-8901-45df-8094-9af7fcd37a8c/manager/0.log" Jul 06 23:27:54.947398 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:54.947373 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84df5575d5-rljrb_5e2c9b14-c631-4931-8083-95b5b40b24d9/copy-manifests/0.log" Jul 06 23:27:54.953543 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:54.953520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84df5575d5-rljrb_5e2c9b14-c631-4931-8083-95b5b40b24d9/manager/0.log" Jul 06 23:27:56.410632 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:56.410604 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-79z46_8e5ec393-e81e-4156-b9f3-fd75651398ae/manager/0.log" Jul 06 23:27:56.860684 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:56.860610 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m4d6x_7af9668c-1492-4f56-b2cc-5a839e5f1029/limitador/0.log" Jul 06 23:27:57.543836 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:57.543810 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f95589f8-hg2l7_a438d8b7-3d21-4dcb-a59a-d6e2bd024970/kube-auth-proxy/0.log" Jul 06 23:27:58.570639 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:58.570612 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-prcp4_53a5d3dc-3931-47f3-bd71-48ae9dbd4f8d/main/0.log" Jul 06 23:27:58.901575 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:58.901555 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5lmf2_6461e12c-ebf2-4d98-b60c-62904ad2eea0/main/0.log" Jul 06 23:27:59.013516 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:27:59.013490 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-f8bg9_a73cee4b-73fa-4d15-90d6-c87c17a94a4f/main/0.log" Jul 06 23:28:02.699575 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:02.699546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:28:02.764796 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:02.764766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:28:03.415211 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:03.415183 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:28:03.468386 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:03.468358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" limit=1024 Jul 06 23:28:05.695815 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:05.695784 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v7ccm_4a33cb94-f91f-4a25-85d5-850bd2ab71e3/global-pull-secret-syncer/0.log" Jul 06 23:28:05.746560 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:05.746532 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6cj5p_74f6aec2-1b74-43cd-9a5d-840a409c64e0/konnectivity-agent/0.log" Jul 06 23:28:05.851809 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:05.851777 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-52.ec2.internal_a8420a51f9e906ef04e32099f52a2c79/haproxy/0.log" Jul 06 23:28:11.026587 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:11.026562 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-79z46_8e5ec393-e81e-4156-b9f3-fd75651398ae/manager/0.log" Jul 06 23:28:11.289982 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:11.289901 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-m4d6x_7af9668c-1492-4f56-b2cc-5a839e5f1029/limitador/0.log" Jul 06 23:28:12.932391 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:12.932363 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ed0e763-297f-4d07-ab33-505cb26be55d/alertmanager/0.log" Jul 06 23:28:12.957363 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:12.957341 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ed0e763-297f-4d07-ab33-505cb26be55d/config-reloader/0.log" Jul 06 23:28:12.979544 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:12.979520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ed0e763-297f-4d07-ab33-505cb26be55d/kube-rbac-proxy-web/0.log" Jul 06 23:28:13.000161 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.000087 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ed0e763-297f-4d07-ab33-505cb26be55d/kube-rbac-proxy/0.log" Jul 06 23:28:13.022522 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.022502 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ed0e763-297f-4d07-ab33-505cb26be55d/kube-rbac-proxy-metric/0.log" Jul 06 23:28:13.044351 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.044331 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ed0e763-297f-4d07-ab33-505cb26be55d/prom-label-proxy/0.log" Jul 06 23:28:13.065648 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.065624 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0ed0e763-297f-4d07-ab33-505cb26be55d/init-config-reloader/0.log" Jul 06 23:28:13.151897 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.151871 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gntjg_8aa77dd4-6ea9-4e8d-aea9-a385229b30ab/kube-state-metrics/0.log" Jul 06 23:28:13.175755 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.175729 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gntjg_8aa77dd4-6ea9-4e8d-aea9-a385229b30ab/kube-rbac-proxy-main/0.log" Jul 06 23:28:13.198187 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.198167 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-gntjg_8aa77dd4-6ea9-4e8d-aea9-a385229b30ab/kube-rbac-proxy-self/0.log" Jul 06 23:28:13.227184 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.227149 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7f66d85f8-xlvkl_27b5e783-00bd-4724-995f-89d634c97d62/metrics-server/0.log" Jul 06 23:28:13.270944 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.270887 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-5qw9f_959664d4-bc99-49d9-b3fa-c71f2ea95944/monitoring-plugin/0.log" Jul 06 23:28:13.414337 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.414308 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2cnq_0b90bb90-5d01-4b71-8025-61a1ed6627fd/node-exporter/0.log" Jul 06 23:28:13.436787 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.436764 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2cnq_0b90bb90-5d01-4b71-8025-61a1ed6627fd/kube-rbac-proxy/0.log" Jul 06 23:28:13.455864 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.455835 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n2cnq_0b90bb90-5d01-4b71-8025-61a1ed6627fd/init-textfile/0.log" Jul 06 23:28:13.569565 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.569493 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rpszq_0751e64e-0b6c-4253-b6c3-879f09840d04/kube-rbac-proxy-main/0.log" Jul 06 23:28:13.592539 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.592517 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rpszq_0751e64e-0b6c-4253-b6c3-879f09840d04/kube-rbac-proxy-self/0.log" Jul 06 23:28:13.616909 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.616887 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-rpszq_0751e64e-0b6c-4253-b6c3-879f09840d04/openshift-state-metrics/0.log" Jul 06 23:28:13.655032 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.655010 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_898ed5f6-bdba-464e-907d-2e66784752bc/prometheus/0.log" Jul 06 23:28:13.676944 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.676924 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_898ed5f6-bdba-464e-907d-2e66784752bc/config-reloader/0.log" Jul 06 23:28:13.697774 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.697758 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_898ed5f6-bdba-464e-907d-2e66784752bc/thanos-sidecar/0.log" Jul 06 23:28:13.720345 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.720328 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_898ed5f6-bdba-464e-907d-2e66784752bc/kube-rbac-proxy-web/0.log" Jul 06 23:28:13.741099 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.741079 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_898ed5f6-bdba-464e-907d-2e66784752bc/kube-rbac-proxy/0.log" Jul 06 23:28:13.761537 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.761520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_898ed5f6-bdba-464e-907d-2e66784752bc/kube-rbac-proxy-thanos/0.log" Jul 06 23:28:13.781693 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.781670 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_898ed5f6-bdba-464e-907d-2e66784752bc/init-config-reloader/0.log" Jul 06 23:28:13.815158 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.815137 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dp9qw_81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d/prometheus-operator/0.log" Jul 06 23:28:13.836831 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.836765 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-dp9qw_81aa2d02-f9e8-4c0a-b1cc-4c1f0204730d/kube-rbac-proxy/0.log" Jul 06 23:28:13.862185 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.862160 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-4vvdh_6cd34a40-7027-4e9e-927d-1d5ba2916c06/prometheus-operator-admission-webhook/0.log" Jul 06 23:28:13.895623 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.895601 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786bd75985-z9hrf_cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68/telemeter-client/0.log" Jul 06 23:28:13.919699 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.919682 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786bd75985-z9hrf_cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68/reload/0.log" Jul 06 23:28:13.940654 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.940633 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-786bd75985-z9hrf_cbfa7aeb-39d6-4c9c-aee2-6311fffc2f68/kube-rbac-proxy/0.log" Jul 06 23:28:13.972422 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.972399 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77556fbd49-v8jdm_69d471a2-071d-4716-9402-6bda4d881c6f/thanos-query/0.log" Jul 06 23:28:13.997043 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:13.997011 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77556fbd49-v8jdm_69d471a2-071d-4716-9402-6bda4d881c6f/kube-rbac-proxy-web/0.log" Jul 06 23:28:14.017057 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:14.017038 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77556fbd49-v8jdm_69d471a2-071d-4716-9402-6bda4d881c6f/kube-rbac-proxy/0.log" Jul 06 23:28:14.038900 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:14.038882 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77556fbd49-v8jdm_69d471a2-071d-4716-9402-6bda4d881c6f/prom-label-proxy/0.log" Jul 06 23:28:14.061484 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:14.061449 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77556fbd49-v8jdm_69d471a2-071d-4716-9402-6bda4d881c6f/kube-rbac-proxy-rules/0.log" Jul 06 23:28:14.085255 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:14.085221 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-77556fbd49-v8jdm_69d471a2-071d-4716-9402-6bda4d881c6f/kube-rbac-proxy-metrics/0.log" Jul 06 23:28:16.225257 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.225228 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf"] Jul 06 23:28:16.225646 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.225634 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93e1de61-9e8e-4b42-9ddc-d8390e79f696" containerName="main" Jul 06 23:28:16.225646 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.225647 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93e1de61-9e8e-4b42-9ddc-d8390e79f696" containerName="main" Jul 06 23:28:16.225718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.225653 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a0530bb-f91c-4b27-b552-554cd6002232" containerName="main" Jul 06 23:28:16.225718 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.225658 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a0530bb-f91c-4b27-b552-554cd6002232" containerName="main" Jul 06 23:28:16.225778 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.225725 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a0530bb-f91c-4b27-b552-554cd6002232" containerName="main" Jul 06 23:28:16.225778 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.225734 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93e1de61-9e8e-4b42-9ddc-d8390e79f696" containerName="main" Jul 06 23:28:16.228826 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.228811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.231385 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.231356 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h8t4j\"/\"kube-root-ca.crt\"" Jul 06 23:28:16.231508 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.231399 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h8t4j\"/\"openshift-service-ca.crt\"" Jul 06 23:28:16.232262 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.232250 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h8t4j\"/\"default-dockercfg-csq5b\"" Jul 06 23:28:16.236593 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.236570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf"] Jul 06 23:28:16.323842 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.323805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-lib-modules\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.323842 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.323842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-proc\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.324063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.323879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-podres\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.324063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.323970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwjq\" (UniqueName: \"kubernetes.io/projected/57912df1-3990-4dd2-bc79-cc1988fac50b-kube-api-access-gjwjq\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.324063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.324010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-sys\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424464 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-proc\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424676 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-podres\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424676 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424533 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwjq\" (UniqueName: \"kubernetes.io/projected/57912df1-3990-4dd2-bc79-cc1988fac50b-kube-api-access-gjwjq\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424676 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424544 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-proc\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424676 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424564 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-sys\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424676 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-lib-modules\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424676 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424654 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-podres\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424913 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-sys\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.424913 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.424761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57912df1-3990-4dd2-bc79-cc1988fac50b-lib-modules\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.432369 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.432348 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwjq\" (UniqueName: \"kubernetes.io/projected/57912df1-3990-4dd2-bc79-cc1988fac50b-kube-api-access-gjwjq\") pod \"perf-node-gather-daemonset-9fgvf\" (UID: \"57912df1-3990-4dd2-bc79-cc1988fac50b\") " pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.504044 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.503973 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-4hzkn_bc362fc8-5494-472c-9b01-122d414ab850/download-server/0.log" Jul 06 23:28:16.539078 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.539055 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:16.664392 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:16.664370 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf"] Jul 06 23:28:16.665474 ip-10-0-134-52 kubenswrapper[2579]: W0706 23:28:16.665438 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod57912df1_3990_4dd2_bc79_cc1988fac50b.slice/crio-c6db2a860c0a263b654e08a98b0fcac2982beeefd0cfdb6c24803b1aeda94777 WatchSource:0}: Error finding container c6db2a860c0a263b654e08a98b0fcac2982beeefd0cfdb6c24803b1aeda94777: Status 404 returned error can't find the container with id c6db2a860c0a263b654e08a98b0fcac2982beeefd0cfdb6c24803b1aeda94777 Jul 06 23:28:17.431671 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:17.431635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" event={"ID":"57912df1-3990-4dd2-bc79-cc1988fac50b","Type":"ContainerStarted","Data":"070ec61b755b17702e24ddc22b37fc65f66bd2109f8668f0ff32df716a0ebabe"} Jul 06 23:28:17.432063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:17.431681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" event={"ID":"57912df1-3990-4dd2-bc79-cc1988fac50b","Type":"ContainerStarted","Data":"c6db2a860c0a263b654e08a98b0fcac2982beeefd0cfdb6c24803b1aeda94777"} Jul 06 23:28:17.432063 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:17.431721 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:17.448260 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:17.448214 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" podStartSLOduration=1.448199506 podStartE2EDuration="1.448199506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:28:17.446650485 +0000 UTC m=+2489.582375060" watchObservedRunningTime="2026-07-06 23:28:17.448199506 +0000 UTC m=+2489.583924061" Jul 06 23:28:18.393736 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:18.393709 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2smb_acceb5a1-7065-409a-a770-4787d1a7642a/dns/0.log" Jul 06 23:28:18.413844 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:18.413818 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c2smb_acceb5a1-7065-409a-a770-4787d1a7642a/kube-rbac-proxy/0.log" Jul 06 23:28:18.436736 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:18.436711 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wdrm_b507c59f-0773-4b0f-9d84-7e9aedb5561c/dns-node-resolver/0.log" Jul 06 23:28:19.043087 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:19.043047 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nv2vk_f4f73e02-0d1f-4c9e-bb6d-052c1a075e94/node-ca/0.log" Jul 06 23:28:20.169914 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:20.169884 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f95589f8-hg2l7_a438d8b7-3d21-4dcb-a59a-d6e2bd024970/kube-auth-proxy/0.log" Jul 06 23:28:21.061002 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:21.060975 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5gc9m_f8cf9a80-4a40-43a7-bb15-c66d2f7af8ee/serve-healthcheck-canary/0.log" Jul 06 23:28:21.696995 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:21.696964 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p8q4d_f93834e1-fa53-48d3-a879-b35a940decac/kube-rbac-proxy/0.log" Jul 06 23:28:21.717334 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:21.717311 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p8q4d_f93834e1-fa53-48d3-a879-b35a940decac/exporter/0.log" Jul 06 23:28:21.736438 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:21.736417 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p8q4d_f93834e1-fa53-48d3-a879-b35a940decac/extractor/0.log" Jul 06 23:28:23.446632 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:23.446605 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h8t4j/perf-node-gather-daemonset-9fgvf" Jul 06 23:28:23.908619 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:23.908591 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-gwnw8_551ce188-8901-45df-8094-9af7fcd37a8c/manager/0.log" Jul 06 23:28:24.498299 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:24.498269 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84df5575d5-rljrb_5e2c9b14-c631-4931-8083-95b5b40b24d9/manager/0.log" Jul 06 23:28:24.534456 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:24.534433 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84df5575d5-rljrb_5e2c9b14-c631-4931-8083-95b5b40b24d9/copy-manifests/0.log" Jul 06 23:28:25.936790 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:25.936766 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5747d5c666-wkdl7_d8f97de7-d850-40fd-9163-998e29347da7/manager/0.log" Jul 06 23:28:25.964453 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:25.964431 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tt92s_ff051f1a-244d-4eba-a642-c31b8249119d/openshift-lws-operator/0.log" Jul 06 23:28:27.135998 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:27.135972 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f95589f8-hg2l7_a438d8b7-3d21-4dcb-a59a-d6e2bd024970/kube-auth-proxy/0.log" Jul 06 23:28:28.633517 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:28.633493 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f95589f8-hg2l7_a438d8b7-3d21-4dcb-a59a-d6e2bd024970/kube-auth-proxy/0.log" Jul 06 23:28:32.666253 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:32.666223 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9cppd_c4f7345a-0a74-4474-9ef1-d611034d3682/kube-multus/0.log" Jul 06 23:28:32.978100 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:32.978027 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phs26_a858fa8c-dc0c-4990-b370-513310efb011/kube-multus-additional-cni-plugins/0.log" Jul 06 23:28:32.998195 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:32.998164 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phs26_a858fa8c-dc0c-4990-b370-513310efb011/egress-router-binary-copy/0.log" Jul 06 23:28:33.018285 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:33.018264 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phs26_a858fa8c-dc0c-4990-b370-513310efb011/cni-plugins/0.log" Jul 06 23:28:33.038809 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:33.038788 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phs26_a858fa8c-dc0c-4990-b370-513310efb011/bond-cni-plugin/0.log" Jul 06 23:28:33.058854 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:33.058834 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phs26_a858fa8c-dc0c-4990-b370-513310efb011/routeoverride-cni/0.log" Jul 06 23:28:33.078996 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:33.078974 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phs26_a858fa8c-dc0c-4990-b370-513310efb011/whereabouts-cni-bincopy/0.log" Jul 06 23:28:33.098521 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:33.098498 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-phs26_a858fa8c-dc0c-4990-b370-513310efb011/whereabouts-cni/0.log" Jul 06 23:28:33.276485 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:33.276411 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cmkgb_efadaccb-16ab-4bf9-aa7d-93db206c15ab/network-metrics-daemon/0.log" Jul 06 23:28:33.297349 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:33.297323 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cmkgb_efadaccb-16ab-4bf9-aa7d-93db206c15ab/kube-rbac-proxy/0.log" Jul 06 23:28:34.148419 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:34.148387 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-controller/0.log" Jul 06 23:28:34.165209 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:34.165183 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/1.log" Jul 06 23:28:34.165209 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:34.165209 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/ovn-acl-logging/0.log" Jul 06 23:28:34.185691 ip-10-0-134-52 kubenswrapper[2579]: I0706 23:28:34.185668 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw68r_e689f81a-bbf3-4c0b-8b1b-86b6c34f394f/kube-rbac-proxy-node/0.log"