Jul 06 15:33:50.950461 ip-10-0-131-241 systemd[1]: Starting Kubernetes Kubelet... Jul 06 15:33:51.332081 ip-10-0-131-241 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:33:51.332081 ip-10-0-131-241 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 15:33:51.332081 ip-10-0-131-241 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:33:51.332081 ip-10-0-131-241 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 15:33:51.332081 ip-10-0-131-241 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 15:33:51.332944 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.332858 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 15:33:51.337759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337740 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:33:51.337759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337758 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:33:51.337759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337762 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337765 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337770 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337772 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337775 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337778 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337781 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337783 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337786 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337788 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337791 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337794 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337796 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337799 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337801 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337804 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337806 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337809 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337817 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337820 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:33:51.337866 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337822 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337826 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337829 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337831 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337834 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337836 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337839 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337842 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337844 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337848 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337850 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337853 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337857 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337861 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337865 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337868 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337872 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337875 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337878 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337882 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:33:51.338353 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337884 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337887 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337890 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337892 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337895 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337898 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337900 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337904 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337908 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337910 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337912 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337915 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337917 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337920 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337922 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337925 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337927 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337930 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337932 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:33:51.338836 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337934 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337939 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337942 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337945 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337948 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337951 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337954 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337957 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337960 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337963 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337966 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337968 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337971 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337974 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337976 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337979 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337981 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337984 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337986 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:33:51.339313 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337988 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337991 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337994 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337996 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.337999 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338001 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338003 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338006 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338008 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338011 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338013 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338016 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338018 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338021 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338023 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338025 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338028 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338030 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338033 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:33:51.339766 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338770 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338778 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338782 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338785 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338789 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338792 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338795 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338798 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338800 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338803 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338806 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338808 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338811 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338813 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338816 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338819 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338821 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338824 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338826 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:33:51.340241 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338829 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338831 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338833 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338836 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338839 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338841 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338844 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338846 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338849 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338851 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338854 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338856 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338858 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338861 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338865 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338869 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338872 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338875 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338877 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338880 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:33:51.340709 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338883 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338885 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338888 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338890 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338893 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338895 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338898 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338900 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338903 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338905 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338908 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338910 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338913 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338915 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338917 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338921 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338924 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338928 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338931 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:33:51.341204 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338934 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338937 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338939 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338942 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338945 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338947 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338950 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338954 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338957 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338959 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338962 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338965 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338967 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338971 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338973 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338976 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338978 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338981 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338984 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338986 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:33:51.341666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338988 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338991 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338993 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338996 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.338998 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339002 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339005 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339008 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339011 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339013 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339016 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339019 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339021 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339024 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339027 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339031 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339033 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339036 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339039 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339042 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:33:51.342154 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339044 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339458 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339465 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339470 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339473 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339477 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339480 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339483 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339486 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339489 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339492 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339495 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339498 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339502 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339506 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339509 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339512 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339514 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339521 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:33:51.342692 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339523 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339526 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339529 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339531 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339533 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339536 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339539 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339543 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339546 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339549 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339551 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339554 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339556 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339559 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339562 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339564 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339566 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339569 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339572 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339575 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:33:51.343172 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339577 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339580 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339582 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339584 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339587 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339589 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339592 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339595 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339597 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339599 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339602 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339606 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339608 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339611 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339614 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339616 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339619 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339623 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339626 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339629 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:33:51.343666 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339631 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339634 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339637 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339639 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339642 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339645 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339648 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339650 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339653 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339655 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339658 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339660 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339664 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339666 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339669 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339671 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339674 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339676 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339679 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:33:51.344208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339682 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339684 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339686 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339689 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339693 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339695 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339698 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339700 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339703 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339705 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339708 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339710 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339713 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339715 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339718 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339720 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339723 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339725 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339728 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339730 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:33:51.344672 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339732 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.339735 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339802 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339810 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339816 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339821 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339825 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339828 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339833 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339837 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339841 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339844 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339848 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339851 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339855 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339858 2575 flags.go:64] FLAG: --cgroup-root="" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339861 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339864 2575 flags.go:64] FLAG: --client-ca-file="" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339867 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339870 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339874 2575 flags.go:64] FLAG: --cluster-domain="" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339877 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339880 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339883 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 15:33:51.345203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339886 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339890 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339893 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339896 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339899 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339902 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339905 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339908 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339910 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339913 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339917 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339920 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339923 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339925 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339928 2575 flags.go:64] FLAG: --enable-server="true" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339932 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339936 2575 flags.go:64] FLAG: --event-burst="100" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339939 2575 flags.go:64] FLAG: --event-qps="50" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339942 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339945 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339948 2575 flags.go:64] FLAG: --eviction-hard="" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339952 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339954 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339957 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339960 2575 flags.go:64] FLAG: --eviction-soft="" Jul 06 15:33:51.345887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339963 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339966 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339969 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339972 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339975 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339978 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339980 2575 flags.go:64] FLAG: --feature-gates="" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339984 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339987 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339990 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339993 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339996 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.339999 2575 flags.go:64] FLAG: --help="false" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340001 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340004 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340007 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340010 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340013 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340016 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340019 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340022 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340024 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340027 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 15:33:51.346511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340031 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340034 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340036 2575 flags.go:64] FLAG: --kube-reserved="" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340039 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340042 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340045 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340048 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340050 2575 flags.go:64] FLAG: --lock-file="" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340053 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340056 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340059 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340064 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340067 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340070 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340072 2575 flags.go:64] FLAG: --logging-format="text" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340075 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340078 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340081 2575 flags.go:64] FLAG: --manifest-url="" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340084 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340088 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340091 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340095 2575 flags.go:64] FLAG: --max-pods="110" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340098 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340101 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340103 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 15:33:51.347124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340106 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340109 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340112 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340115 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340122 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340125 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340127 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340130 2575 flags.go:64] FLAG: --pod-cidr="" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340134 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340139 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340144 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340147 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340150 2575 flags.go:64] FLAG: --port="10250" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340153 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340156 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fbfe578282f96703" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340172 2575 flags.go:64] FLAG: --qos-reserved="" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340175 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340178 2575 flags.go:64] FLAG: --register-node="true" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340181 2575 flags.go:64] FLAG: --register-with-taints="" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340184 2575 flags.go:64] FLAG: --registry-burst="10" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340187 2575 flags.go:64] FLAG: --registry-qps="5" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340190 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340193 2575 flags.go:64] FLAG: --reserved-memory="" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340196 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340199 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 15:33:51.347753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340202 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340205 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340208 2575 flags.go:64] FLAG: --runonce="false" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340210 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340213 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340216 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340218 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340221 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340224 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340227 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340230 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340232 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340235 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340238 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340240 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340243 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340247 2575 flags.go:64] FLAG: --system-cgroups="" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340251 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340257 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340259 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340262 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340266 2575 flags.go:64] FLAG: --tls-min-version="" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340268 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340273 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340275 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 15:33:51.348377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340278 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340281 2575 flags.go:64] FLAG: --v="2" Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340285 2575 flags.go:64] FLAG: --version="false" Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340289 2575 flags.go:64] FLAG: --vmodule="" Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340292 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.340295 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340409 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340412 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340415 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340418 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340420 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340423 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340426 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340428 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340431 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340434 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340436 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340439 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340441 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340444 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340446 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:33:51.349017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340449 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340451 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340454 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340458 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340461 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340464 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340466 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340469 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340472 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340476 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340480 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340484 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340486 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340489 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340492 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340495 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340497 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340500 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340502 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:33:51.349543 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340505 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340508 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340512 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340514 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340517 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340519 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340522 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340524 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340527 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340529 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340532 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340534 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340537 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340539 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340542 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340546 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340551 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340554 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340557 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:33:51.350017 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340560 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340562 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340565 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340569 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340571 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340573 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340576 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340579 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340581 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340584 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340586 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340588 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340591 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340593 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340596 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340598 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340601 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340603 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340605 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340608 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:33:51.350759 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340610 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340613 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340616 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340618 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340620 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340623 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340625 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340628 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340630 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340634 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340637 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340639 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340642 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340644 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340647 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340650 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340653 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340655 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340658 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:33:51.351432 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340660 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340662 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340665 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340667 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340670 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340672 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.340675 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.341652 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.348379 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.348396 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.348413 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.348424 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348481 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348486 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348489 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:33:51.351897 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348493 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348495 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348498 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348500 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348503 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348506 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348508 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348511 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348513 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348516 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348519 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348521 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348524 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348526 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348529 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348531 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348534 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348536 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348538 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:33:51.352303 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348540 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348543 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348545 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348548 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348551 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348553 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348556 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348561 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348563 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348566 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348568 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348571 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348573 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348576 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348578 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348580 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348583 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348585 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348588 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348590 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:33:51.352769 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348593 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348595 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348597 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348600 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348602 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348605 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348607 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348610 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348612 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348615 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348617 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348619 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348622 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348624 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348629 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348633 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348636 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348639 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348643 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348647 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:33:51.353305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348652 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348655 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348657 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348660 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348663 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348666 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348668 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348671 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348673 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348676 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348678 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348681 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348683 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348686 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348688 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348691 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348695 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348698 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348700 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:33:51.353799 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348703 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348705 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348708 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348710 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348713 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348716 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348719 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348721 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348724 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348726 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348729 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348732 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348734 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348737 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348740 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348742 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348744 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:33:51.354305 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348747 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.348752 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348887 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348893 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348896 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348898 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348901 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348904 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348907 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348909 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348912 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348915 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348917 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348920 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348922 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 15:33:51.354718 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348924 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348927 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348929 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348932 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348936 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348938 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348941 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348944 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348946 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348949 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348951 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348954 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348956 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348959 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348962 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348964 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348967 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348969 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348972 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 15:33:51.355090 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348974 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348978 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348981 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348984 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348987 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348990 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348993 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348997 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.348999 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349002 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349004 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349007 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349009 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349012 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349014 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349017 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349020 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349023 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349025 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 15:33:51.355723 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349028 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349031 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349034 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349036 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349038 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349041 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349043 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349045 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349048 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349050 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349055 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349058 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349061 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349063 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349066 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349068 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349071 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349073 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349075 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 15:33:51.356218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349078 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349080 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349082 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349085 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349087 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349089 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349092 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349094 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349097 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349100 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349102 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349106 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349109 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349112 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349114 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349117 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349119 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349122 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349124 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349127 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 15:33:51.356680 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349129 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349131 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349134 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349137 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349139 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349142 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349144 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349147 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:51.349149 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.349153 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.349292 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.352653 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.353937 2575 server.go:1019] "Starting client certificate rotation" Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.354035 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 15:33:51.357182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.354067 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 15:33:51.357613 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.356043 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 15:33:51.357613 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.357261 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 15:33:51.371326 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.371306 2575 log.go:25] "Validated CRI v1 runtime API" Jul 06 15:33:51.373300 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.373287 2575 log.go:25] "Validated CRI v1 image API" Jul 06 15:33:51.374116 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.374102 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 15:33:51.377767 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.377746 2575 fs.go:135] Filesystem UUIDs: map[6ee923d7-01ad-4a5b-a34a-1f683cc84f48:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f40f6782-1fce-4f5b-9861-c098162327d9:/dev/nvme0n1p3] Jul 06 15:33:51.377841 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.377765 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 15:33:51.381440 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.381422 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 15:33:51.382849 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.382738 2575 manager.go:216] Machine: {Timestamp:2026-07-06 15:33:51.381607843 +0000 UTC m=+0.316357883 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3110514 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23997a862e973f032524b8cf9e9dd5 SystemUUID:ec23997a-862e-973f-0325-24b8cf9e9dd5 BootID:f31bee0d-021b-493e-afb8-57660470955f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:96:73:03:54:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:96:73:03:54:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:cf:9c:13:c4:ba Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 15:33:51.382849 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.382847 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 15:33:51.382948 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.382928 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 15:33:51.385893 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.385862 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 15:33:51.386031 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.385895 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 15:33:51.386680 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.386670 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 15:33:51.386719 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.386683 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 15:33:51.386719 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.386691 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 15:33:51.387320 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.387310 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 15:33:51.388560 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.388549 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 15:33:51.392252 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392240 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 06 15:33:51.392389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392380 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 15:33:51.392422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392411 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 06 15:33:51.392422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392420 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 15:33:51.392487 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392437 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 15:33:51.392487 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392448 2575 kubelet.go:398] "Adding apiserver pod source" Jul 06 15:33:51.392487 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392457 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 15:33:51.392820 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392809 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 15:33:51.392854 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.392826 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 15:33:51.396205 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.396182 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 15:33:51.397945 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.397929 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 15:33:51.398039 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.397959 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 15:33:51.399070 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399048 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 15:33:51.399070 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399068 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399077 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399083 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399092 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399097 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399105 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399110 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399116 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399121 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399129 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 15:33:51.399141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399134 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 15:33:51.399480 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399157 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 15:33:51.399480 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.399180 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 15:33:51.402465 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.402448 2575 server.go:1268] "Started kubelet" Jul 06 15:33:51.402579 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.402526 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 15:33:51.402635 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.402555 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 15:33:51.402635 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.402608 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 15:33:51.403289 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.403267 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 15:33:51.403394 ip-10-0-131-241 systemd[1]: Started Kubernetes Kubelet. Jul 06 15:33:51.403520 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.403407 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 15:33:51.403520 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.403471 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 15:33:51.404285 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.404264 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 15:33:51.404802 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.404542 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 06 15:33:51.406255 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.406234 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zb2nk" Jul 06 15:33:51.409107 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.409067 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 15:33:51.410002 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.409195 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-241.ec2.internal.18bfbd39553662ae default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-241.ec2.internal,UID:ip-10-0-131-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-241.ec2.internal,},FirstTimestamp:2026-07-06 15:33:51.402418862 +0000 UTC m=+0.337168901,LastTimestamp:2026-07-06 15:33:51.402418862 +0000 UTC m=+0.337168901,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-241.ec2.internal,}" Jul 06 15:33:51.410590 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.410576 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 15:33:51.410650 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.410590 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 15:33:51.410722 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.410705 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:51.411399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411337 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 15:33:51.411399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411359 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 15:33:51.411399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411364 2575 factory.go:55] Registering systemd factory Jul 06 15:33:51.411399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411335 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 15:33:51.411399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411379 2575 factory.go:223] Registration of the systemd container factory successfully Jul 06 15:33:51.411606 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411451 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 06 15:33:51.411606 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411458 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 06 15:33:51.411606 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411593 2575 factory.go:153] Registering CRI-O factory Jul 06 15:33:51.411606 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411607 2575 factory.go:223] Registration of the crio container factory successfully Jul 06 15:33:51.411775 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411654 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 15:33:51.411775 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411673 2575 factory.go:103] Registering Raw factory Jul 06 15:33:51.411775 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411687 2575 manager.go:1196] Started watching for new ooms in manager Jul 06 15:33:51.412063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.411989 2575 manager.go:349] Starting recovery of all containers Jul 06 15:33:51.414860 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.414834 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zb2nk" Jul 06 15:33:51.421186 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.420180 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 15:33:51.421186 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.420208 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 15:33:51.421649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.421490 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 15:33:51.423427 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.423219 2575 manager.go:354] Recovery completed Jul 06 15:33:51.428535 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.428519 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:33:51.430995 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.430979 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:33:51.431067 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.431008 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:33:51.431067 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.431017 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:33:51.431532 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.431515 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 15:33:51.431532 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.431530 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:33:51.431647 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.431532 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 15:33:51.431647 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.431562 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 06 15:33:51.432192 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.432177 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:33:51.432233 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.432205 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:33:51.432233 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.432215 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:33:51.433691 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.433679 2575 policy_none.go:49] "None policy: Start" Jul 06 15:33:51.433739 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.433694 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 15:33:51.433739 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.433704 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 15:33:51.435665 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.435655 2575 policy_none.go:47] "Start" Jul 06 15:33:51.471804 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.471783 2575 manager.go:337] "Starting Device Plugin manager" Jul 06 15:33:51.471893 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.471828 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 15:33:51.471893 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.471843 2575 server.go:87] "Starting device plugin registration server" Jul 06 15:33:51.472114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.472100 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 15:33:51.472188 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.472120 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 15:33:51.472249 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.472221 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 15:33:51.472355 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.472335 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 15:33:51.472355 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.472352 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 15:33:51.472931 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.472912 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 15:33:51.472993 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.472956 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:51.572530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.572493 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:33:51.574401 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.573697 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:33:51.574401 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.573727 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:33:51.574401 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.573737 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:33:51.574401 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.573764 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.581705 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.581688 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.581753 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.581716 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-241.ec2.internal\": node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:51.596756 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.596696 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 15:33:51.596756 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.596733 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 15:33:51.596756 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.596755 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 15:33:51.596933 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.596784 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 15:33:51.599155 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.599136 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:33:51.616468 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.616439 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:51.697867 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.697838 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal"] Jul 06 15:33:51.697978 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.697919 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:33:51.699611 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.699596 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:33:51.699684 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.699626 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:33:51.699684 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.699637 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:33:51.701641 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.701622 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:33:51.701758 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.701735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.701793 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.701764 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:33:51.702356 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.702338 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:33:51.702450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.702372 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:33:51.702450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.702383 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:33:51.702450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.702337 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:33:51.702450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.702446 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:33:51.702569 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.702469 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:33:51.704521 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.704505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.704609 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.704527 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 15:33:51.705133 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.705118 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientMemory" Jul 06 15:33:51.705235 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.705143 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 15:33:51.705235 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.705152 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeHasSufficientPID" Jul 06 15:33:51.713661 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.713633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c175717ba8b5e711b25b7ec202a4efc9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal\" (UID: \"c175717ba8b5e711b25b7ec202a4efc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.713737 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.713664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c175717ba8b5e711b25b7ec202a4efc9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal\" (UID: \"c175717ba8b5e711b25b7ec202a4efc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.713737 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.713680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c079fdaed824bcba1a1cdfbf52df0d2c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-241.ec2.internal\" (UID: \"c079fdaed824bcba1a1cdfbf52df0d2c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.716727 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.716712 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:51.727825 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.727798 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-241.ec2.internal\" not found" node="ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.732272 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.732254 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-241.ec2.internal\" not found" node="ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.814800 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.814768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c175717ba8b5e711b25b7ec202a4efc9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal\" (UID: \"c175717ba8b5e711b25b7ec202a4efc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.814920 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.814815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c175717ba8b5e711b25b7ec202a4efc9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal\" (UID: \"c175717ba8b5e711b25b7ec202a4efc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.814920 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.814855 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c175717ba8b5e711b25b7ec202a4efc9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal\" (UID: \"c175717ba8b5e711b25b7ec202a4efc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.814920 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.814893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c079fdaed824bcba1a1cdfbf52df0d2c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-241.ec2.internal\" (UID: \"c079fdaed824bcba1a1cdfbf52df0d2c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.815018 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.814945 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c175717ba8b5e711b25b7ec202a4efc9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal\" (UID: \"c175717ba8b5e711b25b7ec202a4efc9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.815018 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:51.814953 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c079fdaed824bcba1a1cdfbf52df0d2c-config\") pod \"kube-apiserver-proxy-ip-10-0-131-241.ec2.internal\" (UID: \"c079fdaed824bcba1a1cdfbf52df0d2c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" Jul 06 15:33:51.816821 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.816805 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:51.917820 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:51.917744 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.018789 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:52.018746 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.030916 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.030895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:52.035682 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.035566 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" Jul 06 15:33:52.118979 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:52.118947 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.219268 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:52.219156 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.320292 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:52.320244 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.353608 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.353579 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 15:33:52.354293 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.353729 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 15:33:52.418436 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.418400 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 15:28:51 +0000 UTC" deadline="2027-12-11 04:10:53.332254753 +0000 UTC" Jul 06 15:33:52.418436 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.418430 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12540h37m0.91382811s" Jul 06 15:33:52.420484 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:52.420465 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.508714 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.508627 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:33:52.521087 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:52.521061 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.621519 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:52.621485 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-241.ec2.internal\" not found" Jul 06 15:33:52.720177 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.720144 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:33:52.723325 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:52.723294 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc079fdaed824bcba1a1cdfbf52df0d2c.slice/crio-3c9c36e0197cbd7758edae912276cdb41921821151c5ced1e54128d981f05161 WatchSource:0}: Error finding container 3c9c36e0197cbd7758edae912276cdb41921821151c5ced1e54128d981f05161: Status 404 returned error can't find the container with id 3c9c36e0197cbd7758edae912276cdb41921821151c5ced1e54128d981f05161 Jul 06 15:33:52.728043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.728029 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:33:52.816155 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.816119 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" Jul 06 15:33:52.831456 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.831429 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 15:33:52.833098 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.833083 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" Jul 06 15:33:52.843299 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.843278 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 15:33:52.901787 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:52.901748 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:33:53.241259 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:53.241228 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc175717ba8b5e711b25b7ec202a4efc9.slice/crio-70a1e32e7e7a3e1bbd21ac3c9887a82c0e20dee505a1282a2317e3b41055319f WatchSource:0}: Error finding container 70a1e32e7e7a3e1bbd21ac3c9887a82c0e20dee505a1282a2317e3b41055319f: Status 404 returned error can't find the container with id 70a1e32e7e7a3e1bbd21ac3c9887a82c0e20dee505a1282a2317e3b41055319f Jul 06 15:33:53.327595 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.327558 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 15:33:53.395623 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.395584 2575 apiserver.go:52] "Watching apiserver" Jul 06 15:33:53.409274 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.409236 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 15:33:53.409671 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.409644 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-45vqq","openshift-network-operator/iptables-alerter-zdsnf","openshift-ovn-kubernetes/ovnkube-node-qlr26","kube-system/global-pull-secret-syncer-jtj2c","kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal","openshift-dns/node-resolver-spgk5","openshift-image-registry/node-ca-q9xfl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal","openshift-multus/multus-additional-cni-plugins-46fm2","openshift-multus/multus-n45dc","openshift-network-diagnostics/network-check-target-2446q","kube-system/konnectivity-agent-8nfwc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc","openshift-cluster-node-tuning-operator/tuned-qg7n5"] Jul 06 15:33:53.412205 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.412161 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 15:33:53.412527 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.412506 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.415615 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.415595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:53.415725 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.415663 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:33:53.416876 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.416853 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 15:33:53.417428 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.417406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.419845 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.419829 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.419924 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.419892 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:33:53.420938 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.420921 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ckwgv\"" Jul 06 15:33:53.421541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421326 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wshd7\"" Jul 06 15:33:53.421541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421352 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 15:33:53.421541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421323 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 15:33:53.421541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421361 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 15:33:53.421541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421425 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 15:33:53.421541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421358 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 15:33:53.421881 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421668 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 15:33:53.421881 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421789 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 15:33:53.421881 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.421854 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 15:33:53.422751 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.422735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.423374 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-node-log\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423471 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-slash\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423471 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-run-netns\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423471 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423439 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-systemd\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423471 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-cni-bin\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423515 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-etc-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423539 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-ovn\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423595 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-env-overrides\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75b8367a-7028-4afe-8603-838a828c77f2-ovn-node-metrics-cert\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.423717 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-ovnkube-script-lib\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.424063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-systemd-units\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.424063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-var-lib-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.424063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-kubelet\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.424063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-log-socket\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.424063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-cni-netd\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.424063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-ovnkube-config\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.424063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.423957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hgz\" (UniqueName: \"kubernetes.io/projected/75b8367a-7028-4afe-8603-838a828c77f2-kube-api-access-24hgz\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.425199 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.425159 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:53.425299 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.425263 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:33:53.428171 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.428151 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.428422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.428401 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 15:33:53.428837 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.428818 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 15:33:53.428934 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.428865 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d2xmv\"" Jul 06 15:33:53.430198 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.430163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.432632 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.432611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.433552 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.433520 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 15:33:53.433634 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.433538 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 15:33:53.434852 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.434834 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zgr8v\"" Jul 06 15:33:53.435235 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.435214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.435661 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.435638 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 15:33:53.436163 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.436145 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:33:53.436163 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.436158 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wrp4m\"" Jul 06 15:33:53.436632 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.436615 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-82k6p\"" Jul 06 15:33:53.437954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.437936 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 15:33:53.438445 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.438427 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.441209 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.441187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.442080 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.442059 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 15:33:53.444988 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.444959 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 15:33:53.445130 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.445106 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 15:33:53.445240 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.445208 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 15:33:53.446598 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.446576 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:33:53.447001 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.446984 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lf7jx\"" Jul 06 15:33:53.447177 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.447163 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 15:33:53.447373 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.447322 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 15:33:53.447497 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.447478 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qp8dh\"" Jul 06 15:33:53.447808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.447787 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 15:33:53.447886 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.447851 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 15:33:53.448362 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.448339 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 15:33:53.448474 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.448454 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rvrd2\"" Jul 06 15:33:53.448614 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.448581 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 15:33:53.491619 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.491546 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 15:33:53.501014 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.500986 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h526f" Jul 06 15:33:53.512469 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.512440 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 15:33:53.522288 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.522259 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h526f" Jul 06 15:33:53.524413 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a990e587-21da-439e-bc4c-7e785b8e6be9-konnectivity-ca\") pod \"konnectivity-agent-8nfwc\" (UID: \"a990e587-21da-439e-bc4c-7e785b8e6be9\") " pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.524547 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysconfig\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.524547 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.524547 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.524681 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524561 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-cnibin\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.524681 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524600 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-kubelet-config\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.524681 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.524681 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cni-binary-copy\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.524681 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.524867 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524696 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-run-netns\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.524867 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.524867 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524798 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-run-ovn-kubernetes\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.524867 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524828 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-run-netns\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.524867 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-cni-bin\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-conf-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.525065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524940 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-daemon-config\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.525065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524953 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-cni-bin\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-socket-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.525065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.524990 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-device-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.525065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-kubernetes\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525102 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-env-overrides\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525128 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75b8367a-7028-4afe-8603-838a828c77f2-ovn-node-metrics-cert\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-tmp-dir\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525332 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-host-slash\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysctl-conf\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.525364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fad70db0-1d09-46ad-a825-e4511534a11a-etc-tuned\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-serviceca\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-systemd-units\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-os-release\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-system-cni-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-os-release\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525512 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-cni-multus\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525526 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-systemd-units\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-run\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77bt\" (UniqueName: \"kubernetes.io/projected/fad70db0-1d09-46ad-a825-e4511534a11a-kube-api-access-f77bt\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525620 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-cni-netd\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525628 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-iptables-alerter-script\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525699 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-cni-netd\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.525742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525743 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-env-overrides\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysctl-d\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzmct\" (UniqueName: \"kubernetes.io/projected/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-kube-api-access-zzmct\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nzx\" (UniqueName: \"kubernetes.io/projected/5d7e1678-72d8-4306-935b-a89536d24248-kube-api-access-n2nzx\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525893 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ffq\" (UniqueName: \"kubernetes.io/projected/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-kube-api-access-k6ffq\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525918 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-cni-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e030cdb6-9045-474d-a6ed-d31ed4293c76-cni-binary-copy\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-kubelet\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.525994 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-etc-kubernetes\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526028 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-registration-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-modprobe-d\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-systemd\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-hosts-file\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a990e587-21da-439e-bc4c-7e785b8e6be9-agent-certs\") pod \"konnectivity-agent-8nfwc\" (UID: \"a990e587-21da-439e-bc4c-7e785b8e6be9\") " pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-sys-fs\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.526389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-sys\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526215 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-host\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-log-socket\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hgz\" (UniqueName: \"kubernetes.io/projected/75b8367a-7028-4afe-8603-838a828c77f2-kube-api-access-24hgz\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cnibin\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526345 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-log-socket\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526396 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc222\" (UniqueName: \"kubernetes.io/projected/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-kube-api-access-pc222\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526456 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-lib-modules\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526482 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcrt\" (UniqueName: \"kubernetes.io/projected/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-kube-api-access-mhcrt\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-node-log\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526757 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-node-log\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-slash\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526840 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-slash\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.527232 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-systemd\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526912 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx5zp\" (UniqueName: \"kubernetes.io/projected/cbbc2574-819d-4aa9-bfa2-91703be68dc5-kube-api-access-vx5zp\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-etc-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-systemd\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527010 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-etc-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.526994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-ovn\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527056 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-run-ovn\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-netns\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-hostroot\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-var-lib-kubelet\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad70db0-1d09-46ad-a825-e4511534a11a-tmp\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-var-lib-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-dbus\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-kubelet\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-ovnkube-config\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527306 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-var-lib-openvswitch\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-system-cni-dir\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527325 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75b8367a-7028-4afe-8603-838a828c77f2-host-kubelet\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-host\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-socket-dir-parent\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527398 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-ovnkube-script-lib\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-k8s-cni-cncf-io\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-cni-bin\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-multus-certs\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rrt8\" (UniqueName: \"kubernetes.io/projected/e030cdb6-9045-474d-a6ed-d31ed4293c76-kube-api-access-6rrt8\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527819 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-ovnkube-config\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.528919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.527948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75b8367a-7028-4afe-8603-838a828c77f2-ovnkube-script-lib\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.529866 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.529846 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75b8367a-7028-4afe-8603-838a828c77f2-ovn-node-metrics-cert\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.545493 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.545462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hgz\" (UniqueName: \"kubernetes.io/projected/75b8367a-7028-4afe-8603-838a828c77f2-kube-api-access-24hgz\") pod \"ovnkube-node-qlr26\" (UID: \"75b8367a-7028-4afe-8603-838a828c77f2\") " pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.602182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.602105 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" event={"ID":"c175717ba8b5e711b25b7ec202a4efc9","Type":"ContainerStarted","Data":"70a1e32e7e7a3e1bbd21ac3c9887a82c0e20dee505a1282a2317e3b41055319f"} Jul 06 15:33:53.603393 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.603363 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" event={"ID":"c079fdaed824bcba1a1cdfbf52df0d2c","Type":"ContainerStarted","Data":"3c9c36e0197cbd7758edae912276cdb41921821151c5ced1e54128d981f05161"} Jul 06 15:33:53.627926 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.627891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-socket-dir-parent\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.627937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-k8s-cni-cncf-io\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.627961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-cni-bin\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.627985 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-multus-certs\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-k8s-cni-cncf-io\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628003 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-socket-dir-parent\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628011 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rrt8\" (UniqueName: \"kubernetes.io/projected/e030cdb6-9045-474d-a6ed-d31ed4293c76-kube-api-access-6rrt8\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628070 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a990e587-21da-439e-bc4c-7e785b8e6be9-konnectivity-ca\") pod \"konnectivity-agent-8nfwc\" (UID: \"a990e587-21da-439e-bc4c-7e785b8e6be9\") " pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-multus-certs\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysconfig\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysconfig\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628070 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-cni-bin\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-cnibin\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-kubelet-config\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cni-binary-copy\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-conf-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628367 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-daemon-config\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628419 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-cnibin\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628423 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-socket-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628467 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-device-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.628480 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628496 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-kubernetes\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.628649 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628526 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-socket-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628537 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-tmp-dir\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628371 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-kubelet-config\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.628592 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret podName:2b0126b9-095c-4bbb-a0b6-a9c2e9359993 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:54.128542807 +0000 UTC m=+3.063292838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret") pod "global-pull-secret-syncer-jtj2c" (UID: "2b0126b9-095c-4bbb-a0b6-a9c2e9359993") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628742 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-conf-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628806 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-device-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a990e587-21da-439e-bc4c-7e785b8e6be9-konnectivity-ca\") pod \"konnectivity-agent-8nfwc\" (UID: \"a990e587-21da-439e-bc4c-7e785b8e6be9\") " pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628894 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628901 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-tmp-dir\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-host-slash\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysctl-conf\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fad70db0-1d09-46ad-a825-e4511534a11a-etc-tuned\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.628968 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-kubernetes\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629004 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-host-slash\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-serviceca\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-os-release\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-system-cni-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.629529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-os-release\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629094 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cni-binary-copy\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629117 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-cni-multus\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-run\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629176 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-os-release\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77bt\" (UniqueName: \"kubernetes.io/projected/fad70db0-1d09-46ad-a825-e4511534a11a-kube-api-access-f77bt\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629196 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-os-release\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-iptables-alerter-script\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629225 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-system-cni-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629246 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-cni-multus\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629095 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysctl-conf\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629278 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-daemon-config\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629286 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629339 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-run\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysctl-d\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.630505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzmct\" (UniqueName: \"kubernetes.io/projected/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-kube-api-access-zzmct\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nzx\" (UniqueName: \"kubernetes.io/projected/5d7e1678-72d8-4306-935b-a89536d24248-kube-api-access-n2nzx\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629507 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ffq\" (UniqueName: \"kubernetes.io/projected/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-kube-api-access-k6ffq\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629514 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-serviceca\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629532 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-cni-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e030cdb6-9045-474d-a6ed-d31ed4293c76-cni-binary-copy\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629579 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-sysctl-d\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629626 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-kubelet\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629582 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-var-lib-kubelet\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-etc-kubernetes\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629698 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-registration-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-modprobe-d\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-systemd\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629820 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-registration-dir\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629854 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-modprobe-d\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629891 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-etc-systemd\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629898 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-etc-kubernetes\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629930 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-hosts-file\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.631303 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.629999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a990e587-21da-439e-bc4c-7e785b8e6be9-agent-certs\") pod \"konnectivity-agent-8nfwc\" (UID: \"a990e587-21da-439e-bc4c-7e785b8e6be9\") " pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-sys-fs\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-sys\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630070 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e030cdb6-9045-474d-a6ed-d31ed4293c76-cni-binary-copy\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630081 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-hosts-file\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630088 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-multus-cni-dir\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630115 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-sys\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630141 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-sys-fs\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630178 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-host\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630215 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-host\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630235 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cnibin\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630264 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630291 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-iptables-alerter-script\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630301 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-cnibin\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc222\" (UniqueName: \"kubernetes.io/projected/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-kube-api-access-pc222\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630350 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-lib-modules\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.630378 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:53.632481 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcrt\" (UniqueName: \"kubernetes.io/projected/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-kube-api-access-mhcrt\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.630439 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs podName:5d7e1678-72d8-4306-935b-a89536d24248 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:54.130422036 +0000 UTC m=+3.065172063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs") pod "network-metrics-daemon-45vqq" (UID: "5d7e1678-72d8-4306-935b-a89536d24248") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630479 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-lib-modules\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx5zp\" (UniqueName: \"kubernetes.io/projected/cbbc2574-819d-4aa9-bfa2-91703be68dc5-kube-api-access-vx5zp\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630568 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cbbc2574-819d-4aa9-bfa2-91703be68dc5-etc-selinux\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630621 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-netns\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-hostroot\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-var-lib-kubelet\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad70db0-1d09-46ad-a825-e4511534a11a-tmp\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-dbus\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630821 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-hostroot\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630832 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-var-lib-kubelet\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630838 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e030cdb6-9045-474d-a6ed-d31ed4293c76-host-run-netns\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-system-cni-dir\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630950 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-system-cni-dir\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630959 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-host\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.631017 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-dbus\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:53.633975 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.630997 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fad70db0-1d09-46ad-a825-e4511534a11a-host\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.633975 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.632816 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fad70db0-1d09-46ad-a825-e4511534a11a-etc-tuned\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.633975 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.632997 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a990e587-21da-439e-bc4c-7e785b8e6be9-agent-certs\") pod \"konnectivity-agent-8nfwc\" (UID: \"a990e587-21da-439e-bc4c-7e785b8e6be9\") " pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.633975 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.633355 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fad70db0-1d09-46ad-a825-e4511534a11a-tmp\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.649896 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.649866 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx5zp\" (UniqueName: \"kubernetes.io/projected/cbbc2574-819d-4aa9-bfa2-91703be68dc5-kube-api-access-vx5zp\") pod \"aws-ebs-csi-driver-node-xx4tc\" (UID: \"cbbc2574-819d-4aa9-bfa2-91703be68dc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.654581 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.654483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzmct\" (UniqueName: \"kubernetes.io/projected/fbcf2182-b539-45cd-bc1b-9fa31f2f5b88-kube-api-access-zzmct\") pod \"node-resolver-spgk5\" (UID: \"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88\") " pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.654984 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.654948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcrt\" (UniqueName: \"kubernetes.io/projected/c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16-kube-api-access-mhcrt\") pod \"node-ca-q9xfl\" (UID: \"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16\") " pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.655922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.655899 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rrt8\" (UniqueName: \"kubernetes.io/projected/e030cdb6-9045-474d-a6ed-d31ed4293c76-kube-api-access-6rrt8\") pod \"multus-n45dc\" (UID: \"e030cdb6-9045-474d-a6ed-d31ed4293c76\") " pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.656596 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.656578 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:33:53.656596 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.656598 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:33:53.656697 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.656609 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jjg22 for pod openshift-network-diagnostics/network-check-target-2446q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:53.656697 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:53.656662 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22 podName:a615b385-ad01-44ed-b97e-3bc14fd4e07b nodeName:}" failed. No retries permitted until 2026-07-06 15:33:54.156647117 +0000 UTC m=+3.091397145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jjg22" (UniqueName: "kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22") pod "network-check-target-2446q" (UID: "a615b385-ad01-44ed-b97e-3bc14fd4e07b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:53.657251 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.657221 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nzx\" (UniqueName: \"kubernetes.io/projected/5d7e1678-72d8-4306-935b-a89536d24248-kube-api-access-n2nzx\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:53.664826 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.664798 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc222\" (UniqueName: \"kubernetes.io/projected/2a8d1e70-c2e7-4383-a4cd-26104219cdbc-kube-api-access-pc222\") pod \"iptables-alerter-zdsnf\" (UID: \"2a8d1e70-c2e7-4383-a4cd-26104219cdbc\") " pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.668628 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.668586 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ffq\" (UniqueName: \"kubernetes.io/projected/a97b91af-ad9a-4b9a-abc4-6c6c56e85cde-kube-api-access-k6ffq\") pod \"multus-additional-cni-plugins-46fm2\" (UID: \"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde\") " pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.671800 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.671780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77bt\" (UniqueName: \"kubernetes.io/projected/fad70db0-1d09-46ad-a825-e4511534a11a-kube-api-access-f77bt\") pod \"tuned-qg7n5\" (UID: \"fad70db0-1d09-46ad-a825-e4511534a11a\") " pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:53.725145 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.725112 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:33:53.734568 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.734543 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q9xfl" Jul 06 15:33:53.742423 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.742366 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-spgk5" Jul 06 15:33:53.751966 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.751936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-46fm2" Jul 06 15:33:53.759449 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.759423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n45dc" Jul 06 15:33:53.767408 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.767381 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zdsnf" Jul 06 15:33:53.775327 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.775286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:33:53.782084 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.782050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" Jul 06 15:33:53.788171 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:53.788131 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" Jul 06 15:33:54.135151 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.135105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:54.135385 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.135181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:54.135385 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.135266 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:54.135385 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.135293 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:54.135385 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.135356 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret podName:2b0126b9-095c-4bbb-a0b6-a9c2e9359993 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:55.135332559 +0000 UTC m=+4.070082592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret") pod "global-pull-secret-syncer-jtj2c" (UID: "2b0126b9-095c-4bbb-a0b6-a9c2e9359993") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:54.135385 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.135379 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs podName:5d7e1678-72d8-4306-935b-a89536d24248 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:55.135366626 +0000 UTC m=+4.070116652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs") pod "network-metrics-daemon-45vqq" (UID: "5d7e1678-72d8-4306-935b-a89536d24248") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:54.235530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.235484 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:54.235762 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.235701 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:33:54.235762 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.235729 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:33:54.235762 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.235743 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jjg22 for pod openshift-network-diagnostics/network-check-target-2446q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:54.235937 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.235814 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22 podName:a615b385-ad01-44ed-b97e-3bc14fd4e07b nodeName:}" failed. No retries permitted until 2026-07-06 15:33:55.235791255 +0000 UTC m=+4.170541282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjg22" (UniqueName: "kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22") pod "network-check-target-2446q" (UID: "a615b385-ad01-44ed-b97e-3bc14fd4e07b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:54.462655 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.462616 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b8367a_7028_4afe_8603_838a828c77f2.slice/crio-0224fec037080af565bc5a58c029eba9d839a081c7434be65d2987a4349028d1 WatchSource:0}: Error finding container 0224fec037080af565bc5a58c029eba9d839a081c7434be65d2987a4349028d1: Status 404 returned error can't find the container with id 0224fec037080af565bc5a58c029eba9d839a081c7434be65d2987a4349028d1 Jul 06 15:33:54.463318 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.463291 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f213b1_bd93_40d0_8ffe_71b3cb4dcd16.slice/crio-3f315a67e23086979b0fa1550508f0ca0dd494f0e88e229eb11a8bf82ab1e5ad WatchSource:0}: Error finding container 3f315a67e23086979b0fa1550508f0ca0dd494f0e88e229eb11a8bf82ab1e5ad: Status 404 returned error can't find the container with id 3f315a67e23086979b0fa1550508f0ca0dd494f0e88e229eb11a8bf82ab1e5ad Jul 06 15:33:54.464479 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.464457 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad70db0_1d09_46ad_a825_e4511534a11a.slice/crio-d6b3227b5f155499fe28d89fe7aa5a259b7cc8ce5f71ac9a9a01cd24439dca12 WatchSource:0}: Error finding container d6b3227b5f155499fe28d89fe7aa5a259b7cc8ce5f71ac9a9a01cd24439dca12: Status 404 returned error can't find the container with id d6b3227b5f155499fe28d89fe7aa5a259b7cc8ce5f71ac9a9a01cd24439dca12 Jul 06 15:33:54.500215 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.500182 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbcf2182_b539_45cd_bc1b_9fa31f2f5b88.slice/crio-a2361c8ee07c91f9619f86e7771f7f2ebc9a6aa8f904243836260d34ec2bc7c1 WatchSource:0}: Error finding container a2361c8ee07c91f9619f86e7771f7f2ebc9a6aa8f904243836260d34ec2bc7c1: Status 404 returned error can't find the container with id a2361c8ee07c91f9619f86e7771f7f2ebc9a6aa8f904243836260d34ec2bc7c1 Jul 06 15:33:54.502518 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.502471 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8d1e70_c2e7_4383_a4cd_26104219cdbc.slice/crio-4a9e0a270b2d3dc67ca192abc689a129224fd4992954e69d299a6e3b19497450 WatchSource:0}: Error finding container 4a9e0a270b2d3dc67ca192abc689a129224fd4992954e69d299a6e3b19497450: Status 404 returned error can't find the container with id 4a9e0a270b2d3dc67ca192abc689a129224fd4992954e69d299a6e3b19497450 Jul 06 15:33:54.503129 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.503016 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbc2574_819d_4aa9_bfa2_91703be68dc5.slice/crio-7780c6a91df4eb35593bbb97f4308e0351cc40a15760a56e0879fdc832616e45 WatchSource:0}: Error finding container 7780c6a91df4eb35593bbb97f4308e0351cc40a15760a56e0879fdc832616e45: Status 404 returned error can't find the container with id 7780c6a91df4eb35593bbb97f4308e0351cc40a15760a56e0879fdc832616e45 Jul 06 15:33:54.504218 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.504187 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda97b91af_ad9a_4b9a_abc4_6c6c56e85cde.slice/crio-adacca2eb70b8ff4a2ae0833288b8dedd95c8288340d7b6605646cc224983152 WatchSource:0}: Error finding container adacca2eb70b8ff4a2ae0833288b8dedd95c8288340d7b6605646cc224983152: Status 404 returned error can't find the container with id adacca2eb70b8ff4a2ae0833288b8dedd95c8288340d7b6605646cc224983152 Jul 06 15:33:54.505545 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:33:54.505520 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda990e587_21da_439e_bc4c_7e785b8e6be9.slice/crio-386c5ac8270734365fd18c0394970e579f26eda4ba769a60532663d247300ed6 WatchSource:0}: Error finding container 386c5ac8270734365fd18c0394970e579f26eda4ba769a60532663d247300ed6: Status 404 returned error can't find the container with id 386c5ac8270734365fd18c0394970e579f26eda4ba769a60532663d247300ed6 Jul 06 15:33:54.525942 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.525901 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:28:53 +0000 UTC" deadline="2028-04-13 11:57:50.856103752 +0000 UTC" Jul 06 15:33:54.525942 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.525938 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15524h23m56.330170363s" Jul 06 15:33:54.597184 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.597139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:54.597347 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.597150 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:54.597347 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.597293 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:33:54.597473 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:54.597356 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:33:54.610109 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.610077 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8nfwc" event={"ID":"a990e587-21da-439e-bc4c-7e785b8e6be9","Type":"ContainerStarted","Data":"386c5ac8270734365fd18c0394970e579f26eda4ba769a60532663d247300ed6"} Jul 06 15:33:54.611485 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.611460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" event={"ID":"cbbc2574-819d-4aa9-bfa2-91703be68dc5","Type":"ContainerStarted","Data":"7780c6a91df4eb35593bbb97f4308e0351cc40a15760a56e0879fdc832616e45"} Jul 06 15:33:54.612724 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.612696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spgk5" event={"ID":"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88","Type":"ContainerStarted","Data":"a2361c8ee07c91f9619f86e7771f7f2ebc9a6aa8f904243836260d34ec2bc7c1"} Jul 06 15:33:54.614187 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.614144 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" event={"ID":"fad70db0-1d09-46ad-a825-e4511534a11a","Type":"ContainerStarted","Data":"d6b3227b5f155499fe28d89fe7aa5a259b7cc8ce5f71ac9a9a01cd24439dca12"} Jul 06 15:33:54.615565 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.615537 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9xfl" event={"ID":"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16","Type":"ContainerStarted","Data":"3f315a67e23086979b0fa1550508f0ca0dd494f0e88e229eb11a8bf82ab1e5ad"} Jul 06 15:33:54.616862 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.616771 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"0224fec037080af565bc5a58c029eba9d839a081c7434be65d2987a4349028d1"} Jul 06 15:33:54.620247 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.620222 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n45dc" event={"ID":"e030cdb6-9045-474d-a6ed-d31ed4293c76","Type":"ContainerStarted","Data":"c5c264935d4dcb83e6e9b1dd39c025ed1fd1d1877a0b271e1f7bef462c590df9"} Jul 06 15:33:54.621565 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.621541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerStarted","Data":"adacca2eb70b8ff4a2ae0833288b8dedd95c8288340d7b6605646cc224983152"} Jul 06 15:33:54.622776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.622751 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdsnf" event={"ID":"2a8d1e70-c2e7-4383-a4cd-26104219cdbc","Type":"ContainerStarted","Data":"4a9e0a270b2d3dc67ca192abc689a129224fd4992954e69d299a6e3b19497450"} Jul 06 15:33:54.636279 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:54.636225 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" podStartSLOduration=1.636188923 podStartE2EDuration="1.636188923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:33:54.636151898 +0000 UTC m=+3.570901950" watchObservedRunningTime="2026-07-06 15:33:54.636188923 +0000 UTC m=+3.570938973" Jul 06 15:33:55.141900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:55.141856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:55.142114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:55.141935 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:55.142114 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.142081 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:55.142232 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.142150 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret podName:2b0126b9-095c-4bbb-a0b6-a9c2e9359993 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:57.142127643 +0000 UTC m=+6.076877676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret") pod "global-pull-secret-syncer-jtj2c" (UID: "2b0126b9-095c-4bbb-a0b6-a9c2e9359993") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:55.142643 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.142623 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:55.142716 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.142685 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs podName:5d7e1678-72d8-4306-935b-a89536d24248 nodeName:}" failed. No retries permitted until 2026-07-06 15:33:57.142668509 +0000 UTC m=+6.077418541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs") pod "network-metrics-daemon-45vqq" (UID: "5d7e1678-72d8-4306-935b-a89536d24248") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:55.243308 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:55.242539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:55.243308 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.242811 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:33:55.243308 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.242832 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:33:55.243308 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.242867 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jjg22 for pod openshift-network-diagnostics/network-check-target-2446q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:55.243308 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.242933 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22 podName:a615b385-ad01-44ed-b97e-3bc14fd4e07b nodeName:}" failed. No retries permitted until 2026-07-06 15:33:57.242914754 +0000 UTC m=+6.177664784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjg22" (UniqueName: "kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22") pod "network-check-target-2446q" (UID: "a615b385-ad01-44ed-b97e-3bc14fd4e07b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:55.526878 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:55.526738 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:28:53 +0000 UTC" deadline="2027-12-20 23:45:30.256051185 +0000 UTC" Jul 06 15:33:55.526878 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:55.526801 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12776h11m34.729254383s" Jul 06 15:33:55.597573 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:55.597538 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:55.597762 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:55.597685 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:33:55.631702 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:55.631085 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-241.ec2.internal" event={"ID":"c079fdaed824bcba1a1cdfbf52df0d2c","Type":"ContainerStarted","Data":"3f2d0daf0ec146be4ef9a8eef77c6f8ea7a0ab5a4a379458c2a4a5f884189fa9"} Jul 06 15:33:56.598324 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:56.597788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:56.598324 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:56.597936 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:33:56.598324 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:56.598320 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:56.598941 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:56.598438 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:33:57.163536 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:57.163439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:57.163536 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:57.163518 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:57.163817 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.163672 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:57.163817 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.163774 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret podName:2b0126b9-095c-4bbb-a0b6-a9c2e9359993 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:01.163753668 +0000 UTC m=+10.098503719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret") pod "global-pull-secret-syncer-jtj2c" (UID: "2b0126b9-095c-4bbb-a0b6-a9c2e9359993") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:33:57.164266 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.164229 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:57.164358 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.164331 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs podName:5d7e1678-72d8-4306-935b-a89536d24248 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:01.164316379 +0000 UTC m=+10.099066412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs") pod "network-metrics-daemon-45vqq" (UID: "5d7e1678-72d8-4306-935b-a89536d24248") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:33:57.264089 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:57.264041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:57.264340 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.264322 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:33:57.264413 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.264352 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:33:57.264413 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.264367 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jjg22 for pod openshift-network-diagnostics/network-check-target-2446q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:57.264533 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.264429 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22 podName:a615b385-ad01-44ed-b97e-3bc14fd4e07b nodeName:}" failed. No retries permitted until 2026-07-06 15:34:01.26441003 +0000 UTC m=+10.199160060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjg22" (UniqueName: "kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22") pod "network-check-target-2446q" (UID: "a615b385-ad01-44ed-b97e-3bc14fd4e07b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:33:57.598968 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:57.598940 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:57.599441 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:57.599047 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:33:58.596982 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:58.596952 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:33:58.597200 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:58.597072 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:33:58.597254 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:58.597219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:33:58.597323 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:58.597300 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:33:59.597808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:33:59.597247 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:33:59.597808 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:33:59.597398 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:00.597489 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:00.597273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:00.597489 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:00.597286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:00.597489 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:00.597430 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:00.597776 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:00.597532 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:01.201564 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:01.201527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:01.201968 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:01.201609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:01.201968 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.201667 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:34:01.201968 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.201695 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:34:01.201968 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.201736 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret podName:2b0126b9-095c-4bbb-a0b6-a9c2e9359993 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:09.20171806 +0000 UTC m=+18.136468086 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret") pod "global-pull-secret-syncer-jtj2c" (UID: "2b0126b9-095c-4bbb-a0b6-a9c2e9359993") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:34:01.201968 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.201758 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs podName:5d7e1678-72d8-4306-935b-a89536d24248 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:09.201746567 +0000 UTC m=+18.136496592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs") pod "network-metrics-daemon-45vqq" (UID: "5d7e1678-72d8-4306-935b-a89536d24248") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:34:01.302806 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:01.302766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:01.302987 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.302918 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:34:01.302987 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.302932 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:34:01.302987 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.302941 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jjg22 for pod openshift-network-diagnostics/network-check-target-2446q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:34:01.302987 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.302985 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22 podName:a615b385-ad01-44ed-b97e-3bc14fd4e07b nodeName:}" failed. No retries permitted until 2026-07-06 15:34:09.302972085 +0000 UTC m=+18.237722114 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjg22" (UniqueName: "kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22") pod "network-check-target-2446q" (UID: "a615b385-ad01-44ed-b97e-3bc14fd4e07b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:34:01.598129 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:01.597995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:01.598129 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:01.598098 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:02.598110 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:02.597603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:02.598110 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:02.597642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:02.598110 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:02.597721 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:02.598110 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:02.597788 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:02.652661 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:02.651976 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" event={"ID":"c175717ba8b5e711b25b7ec202a4efc9","Type":"ContainerStarted","Data":"c559acba31dd24f509a393791706b4d1cabb9b52e74f43ec500fc2995e59a94c"} Jul 06 15:34:03.600936 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:03.599770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:03.600936 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:03.599932 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:04.597825 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:04.597790 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:04.598000 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:04.597823 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:04.598000 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:04.597923 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:04.598100 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:04.598076 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:05.597450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.597414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:05.598007 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:05.597539 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:05.659446 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.659408 2575 generic.go:356] "Generic (PLEG): container finished" podID="a97b91af-ad9a-4b9a-abc4-6c6c56e85cde" containerID="e13cd5111eb2707c5cba331483154f152bd4570dc51e228e92584096a009b0eb" exitCode=0 Jul 06 15:34:05.659637 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.659493 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerDied","Data":"e13cd5111eb2707c5cba331483154f152bd4570dc51e228e92584096a009b0eb"} Jul 06 15:34:05.661089 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.661059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8nfwc" event={"ID":"a990e587-21da-439e-bc4c-7e785b8e6be9","Type":"ContainerStarted","Data":"87115df221fa133ff321070edcc1a62d1e66bc6c223f38e0e0a162bc613afdd5"} Jul 06 15:34:05.662655 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.662630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" event={"ID":"cbbc2574-819d-4aa9-bfa2-91703be68dc5","Type":"ContainerStarted","Data":"db20bcd7cc8f2f88d6e9be5df96c0868ab97889579b82984d71f19b61920b4b6"} Jul 06 15:34:05.664415 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.663985 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-spgk5" event={"ID":"fbcf2182-b539-45cd-bc1b-9fa31f2f5b88","Type":"ContainerStarted","Data":"ff248e0d355907745d39944a4d65155b91b3c44a3f8aad2534d0a60308c55d0a"} Jul 06 15:34:05.674207 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.674146 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" event={"ID":"fad70db0-1d09-46ad-a825-e4511534a11a","Type":"ContainerStarted","Data":"fa2133720dff68e70c24d0c093984e52e16b320c43eb354eb58f8be7920c6f5f"} Jul 06 15:34:05.677692 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.677666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q9xfl" event={"ID":"c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16","Type":"ContainerStarted","Data":"edc4c8954202c754d202b440ebefb5f6830e2b5572f1eff91d7e4f49c0f1bd54"} Jul 06 15:34:05.679285 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.679261 2575 generic.go:356] "Generic (PLEG): container finished" podID="c175717ba8b5e711b25b7ec202a4efc9" containerID="c559acba31dd24f509a393791706b4d1cabb9b52e74f43ec500fc2995e59a94c" exitCode=0 Jul 06 15:34:05.679375 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.679299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" event={"ID":"c175717ba8b5e711b25b7ec202a4efc9","Type":"ContainerDied","Data":"c559acba31dd24f509a393791706b4d1cabb9b52e74f43ec500fc2995e59a94c"} Jul 06 15:34:05.733064 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.732996 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-spgk5" podStartSLOduration=3.386919243 podStartE2EDuration="13.732977167s" totalImagesPullingTime="10.346057924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:05.714124231 +0000 UTC m=+14.648874315" watchObservedRunningTime="2026-07-06 15:34:05.732977167 +0000 UTC m=+14.667727215" Jul 06 15:34:05.733241 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.733100 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q9xfl" podStartSLOduration=3.3708847410000002 podStartE2EDuration="13.733094308s" totalImagesPullingTime="10.362209567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:05.732874569 +0000 UTC m=+14.667624640" watchObservedRunningTime="2026-07-06 15:34:05.733094308 +0000 UTC m=+14.667844356" Jul 06 15:34:05.754362 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.754308 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8nfwc" podStartSLOduration=3.425646609 podStartE2EDuration="13.754288406s" totalImagesPullingTime="10.328641797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:05.753816331 +0000 UTC m=+14.688566403" watchObservedRunningTime="2026-07-06 15:34:05.754288406 +0000 UTC m=+14.689038454" Jul 06 15:34:05.791047 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.791014 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:34:05.791620 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.791596 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:34:05.795153 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:05.795097 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qg7n5" podStartSLOduration=3.436556823 podStartE2EDuration="13.795079991s" totalImagesPullingTime="10.358523168s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:05.778096922 +0000 UTC m=+14.712846970" watchObservedRunningTime="2026-07-06 15:34:05.795079991 +0000 UTC m=+14.729830039" Jul 06 15:34:06.597122 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:06.597095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:06.597341 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:06.597230 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:06.597341 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:06.597286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:06.597445 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:06.597397 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:06.683629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:06.683541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zdsnf" event={"ID":"2a8d1e70-c2e7-4383-a4cd-26104219cdbc","Type":"ContainerStarted","Data":"11adfbf4edd51fe4062d0cc1ad5263a53a9ea7375e9bcaa410d81baa3999134e"} Jul 06 15:34:06.702316 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:06.702245 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zdsnf" podStartSLOduration=4.367118415 podStartE2EDuration="14.702229908s" totalImagesPullingTime="10.335111493s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:06.70159331 +0000 UTC m=+15.636343358" watchObservedRunningTime="2026-07-06 15:34:06.702229908 +0000 UTC m=+15.636979955" Jul 06 15:34:07.597039 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:07.597007 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:07.597224 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:07.597147 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:07.685898 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:07.685872 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 15:34:08.597702 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:08.597660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:08.597886 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:08.597671 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:08.597886 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:08.597770 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:08.597886 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:08.597870 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:09.263674 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:09.263635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:09.264316 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:09.263694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:09.264316 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.263768 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:34:09.264316 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.263812 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:34:09.264316 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.263852 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs podName:5d7e1678-72d8-4306-935b-a89536d24248 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:25.263828258 +0000 UTC m=+34.198578287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs") pod "network-metrics-daemon-45vqq" (UID: "5d7e1678-72d8-4306-935b-a89536d24248") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:34:09.264316 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.263872 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret podName:2b0126b9-095c-4bbb-a0b6-a9c2e9359993 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:25.26386208 +0000 UTC m=+34.198612107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret") pod "global-pull-secret-syncer-jtj2c" (UID: "2b0126b9-095c-4bbb-a0b6-a9c2e9359993") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:34:09.364711 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:09.364674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:09.364885 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.364826 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:34:09.364885 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.364844 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:34:09.364885 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.364853 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jjg22 for pod openshift-network-diagnostics/network-check-target-2446q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:34:09.365075 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.364907 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22 podName:a615b385-ad01-44ed-b97e-3bc14fd4e07b nodeName:}" failed. No retries permitted until 2026-07-06 15:34:25.364889913 +0000 UTC m=+34.299639937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjg22" (UniqueName: "kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22") pod "network-check-target-2446q" (UID: "a615b385-ad01-44ed-b97e-3bc14fd4e07b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:34:09.597811 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:09.597777 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:09.597992 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:09.597910 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:10.597614 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:10.597583 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:10.598006 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:10.597595 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:10.598006 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:10.597722 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:10.598006 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:10.597779 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:11.430888 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:11.430641 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:34:11.431067 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:11.431004 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 15:34:11.431470 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:11.431442 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8nfwc" Jul 06 15:34:11.599942 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:11.599906 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:11.600358 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:11.600031 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:12.597467 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:12.597430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:12.597638 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:12.597434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:12.597638 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:12.597547 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:12.597758 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:12.597636 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:13.597111 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:13.597072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:13.597734 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:13.597231 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:14.597950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:14.597918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:14.597950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:14.597935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:14.598516 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:14.598053 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:14.598516 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:14.598187 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:15.597433 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:15.597390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:15.597618 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:15.597534 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:16.439931 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.439903 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 15:34:16.490820 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.490671 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T15:34:16.439927523Z","UUID":"2936eb66-a825-4cf4-8064-8fba25e91984","Handler":null,"Name":"","Endpoint":""} Jul 06 15:34:16.492201 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.492184 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 15:34:16.492291 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.492220 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 15:34:16.597569 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.597532 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:16.597710 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.597572 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:16.597710 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:16.597678 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:16.597832 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:16.597812 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:16.705821 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.705793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n45dc" event={"ID":"e030cdb6-9045-474d-a6ed-d31ed4293c76","Type":"ContainerStarted","Data":"ccc9164601a69b22c43312fee75a16379df05d01aa71b32b8534974ae7e13577"} Jul 06 15:34:16.708378 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.708355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" event={"ID":"cbbc2574-819d-4aa9-bfa2-91703be68dc5","Type":"ContainerStarted","Data":"2db0b62c45d1a2b440c4fb7a371a333407c95b3906db80739e606aad7d79a210"} Jul 06 15:34:16.710352 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.710324 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"b98c120477af99d1d6d4fbe0ec9a08d2687a63459fb736ae0ce366c2d97f3051"} Jul 06 15:34:16.710451 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.710407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"202c59b7be7415de19ca77b067b8f20200e8d3d4fed901c5aba52897be5555b0"} Jul 06 15:34:16.712339 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.712317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" event={"ID":"c175717ba8b5e711b25b7ec202a4efc9","Type":"ContainerStarted","Data":"47eaa978597b222b2d42235e412bb1d1fe968508ee69cc28354a45353020a727"} Jul 06 15:34:16.730204 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.730120 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n45dc" podStartSLOduration=3.014265896 podStartE2EDuration="24.73009608s" totalImagesPullingTime="21.715830184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:16.729788516 +0000 UTC m=+25.664538565" watchObservedRunningTime="2026-07-06 15:34:16.73009608 +0000 UTC m=+25.664846128" Jul 06 15:34:16.745114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:16.745074 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-241.ec2.internal" podStartSLOduration=23.745057164 podStartE2EDuration="23.745057164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:16.744892804 +0000 UTC m=+25.679642852" watchObservedRunningTime="2026-07-06 15:34:16.745057164 +0000 UTC m=+25.679807210" Jul 06 15:34:17.597691 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:17.597663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:17.598273 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:17.597769 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:17.718184 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:17.718126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"0eecec206295d4bde3864bcd19fb5a4379053b78da57b6449d51fc84cef5d5a9"} Jul 06 15:34:17.718338 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:17.718224 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"e586739abc4d13d0cc7b96c8a2ba806d6401e7db65201193d3a4161b7e54df40"} Jul 06 15:34:17.718338 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:17.718268 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"bdd0ac258e94540e5ebe298d0c8311dc0d719085682d6c69fee2dabb27ce2754"} Jul 06 15:34:17.718338 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:17.718308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"c607da4f0baadba93cdfb8cf4af3d1c487eeb66171d5d830085dd6aede9d6bfe"} Jul 06 15:34:17.722235 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:17.722197 2575 generic.go:356] "Generic (PLEG): container finished" podID="a97b91af-ad9a-4b9a-abc4-6c6c56e85cde" containerID="b8f4f5783101298444d2de2cc624d35235d3426cbe5128e235ce26fa69a5b3ce" exitCode=0 Jul 06 15:34:17.722483 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:17.722453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerDied","Data":"b8f4f5783101298444d2de2cc624d35235d3426cbe5128e235ce26fa69a5b3ce"} Jul 06 15:34:18.597569 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:18.597537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:18.597727 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:18.597651 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:18.597727 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:18.597703 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:18.598138 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:18.597837 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:18.726474 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:18.726440 2575 generic.go:356] "Generic (PLEG): container finished" podID="a97b91af-ad9a-4b9a-abc4-6c6c56e85cde" containerID="1868ff8e7930fbaa3bfff7be1ae8adbf937ea1bfce40bf7cff0b975a059bf9a2" exitCode=0 Jul 06 15:34:18.726650 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:18.726527 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerDied","Data":"1868ff8e7930fbaa3bfff7be1ae8adbf937ea1bfce40bf7cff0b975a059bf9a2"} Jul 06 15:34:18.728574 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:18.728552 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" event={"ID":"cbbc2574-819d-4aa9-bfa2-91703be68dc5","Type":"ContainerStarted","Data":"3e751de612b35eaed8e7adf76193518caa2be3bf13b476b8be4fb27021044a97"} Jul 06 15:34:18.768889 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:18.768772 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xx4tc" podStartSLOduration=4.098284549 podStartE2EDuration="26.768756956s" totalImagesPullingTime="22.670472407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:18.768412518 +0000 UTC m=+27.703162564" watchObservedRunningTime="2026-07-06 15:34:18.768756956 +0000 UTC m=+27.703507003" Jul 06 15:34:19.597638 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:19.597550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:19.597811 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:19.597657 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:19.733876 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:19.733835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"97bd76f63656ac3c36767805707b21a3ff7199c9a6563eb6bc89b73d77bf1ea5"} Jul 06 15:34:19.738191 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:19.736478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerDied","Data":"7a9fcb5bc7944c73d0fe7e50643787b619e6cca500f305e6de785ab37cfb5026"} Jul 06 15:34:19.738191 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:19.736404 2575 generic.go:356] "Generic (PLEG): container finished" podID="a97b91af-ad9a-4b9a-abc4-6c6c56e85cde" containerID="7a9fcb5bc7944c73d0fe7e50643787b619e6cca500f305e6de785ab37cfb5026" exitCode=0 Jul 06 15:34:20.597706 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:20.597673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:20.597884 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:20.597672 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:20.597884 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:20.597801 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:20.598287 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:20.597894 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:21.598675 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:21.598457 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:21.599252 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:21.598750 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:21.746152 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:21.746117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" event={"ID":"75b8367a-7028-4afe-8603-838a828c77f2","Type":"ContainerStarted","Data":"f1aeb249544a59b527352123e101fe6efae47cb7d20997629da19813dced5cb7"} Jul 06 15:34:21.746450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:21.746430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:34:21.746541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:21.746511 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:34:21.765907 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:21.765876 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:34:21.780746 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:21.780679 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" podStartSLOduration=9.020108446 podStartE2EDuration="30.78066037s" totalImagesPullingTime="21.760551924s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:21.78032583 +0000 UTC m=+30.715075897" watchObservedRunningTime="2026-07-06 15:34:21.78066037 +0000 UTC m=+30.715410418" Jul 06 15:34:22.597765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:22.597725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:22.597765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:22.597763 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:22.597975 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:22.597878 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:22.598033 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:22.598011 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:22.748543 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:22.748509 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:34:22.764423 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:22.764395 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:34:23.598910 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:23.598870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:23.599077 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:23.598989 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:23.655157 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:23.655113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jtj2c"] Jul 06 15:34:23.655352 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:23.655253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:23.655418 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:23.655339 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:23.657441 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:23.657416 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-45vqq"] Jul 06 15:34:23.657577 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:23.657510 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:23.657638 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:23.657611 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:23.678405 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:23.678314 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2446q"] Jul 06 15:34:23.751439 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:23.751404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:23.751897 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:23.751531 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:25.280675 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.280494 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:25.281234 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.280677 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:34:25.281234 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.280695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:25.281234 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.280755 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs podName:5d7e1678-72d8-4306-935b-a89536d24248 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:57.280734867 +0000 UTC m=+66.215484920 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs") pod "network-metrics-daemon-45vqq" (UID: "5d7e1678-72d8-4306-935b-a89536d24248") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 15:34:25.281234 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.280785 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 15:34:25.281234 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.280832 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret podName:2b0126b9-095c-4bbb-a0b6-a9c2e9359993 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:57.280817154 +0000 UTC m=+66.215567178 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret") pod "global-pull-secret-syncer-jtj2c" (UID: "2b0126b9-095c-4bbb-a0b6-a9c2e9359993") : object "kube-system"/"original-pull-secret" not registered Jul 06 15:34:25.381995 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.381965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:25.382144 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.382089 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 15:34:25.382144 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.382103 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 15:34:25.382144 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.382113 2575 projected.go:196] Error preparing data for projected volume kube-api-access-jjg22 for pod openshift-network-diagnostics/network-check-target-2446q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:34:25.382272 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.382182 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22 podName:a615b385-ad01-44ed-b97e-3bc14fd4e07b nodeName:}" failed. No retries permitted until 2026-07-06 15:34:57.382147947 +0000 UTC m=+66.316897971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jjg22" (UniqueName: "kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22") pod "network-check-target-2446q" (UID: "a615b385-ad01-44ed-b97e-3bc14fd4e07b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 15:34:25.597552 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.597523 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:25.597723 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.597561 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:25.597723 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.597523 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:25.597723 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.597661 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:25.597892 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.597720 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:25.597892 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:25.597782 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:25.759098 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.759067 2575 generic.go:356] "Generic (PLEG): container finished" podID="a97b91af-ad9a-4b9a-abc4-6c6c56e85cde" containerID="ab0aae07ce183348c9121e7cbd2fa1084b4b0ebc7764ec5052b34ebb4192c705" exitCode=0 Jul 06 15:34:25.759275 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:25.759119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerDied","Data":"ab0aae07ce183348c9121e7cbd2fa1084b4b0ebc7764ec5052b34ebb4192c705"} Jul 06 15:34:26.346914 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:26.346881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:26.456784 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:26.456755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:26.521222 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:26.521189 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:26.764576 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:26.764498 2575 generic.go:356] "Generic (PLEG): container finished" podID="a97b91af-ad9a-4b9a-abc4-6c6c56e85cde" containerID="2be61aca890bc463019913857b941eb819c8608a54205d79d4d4c082b5472f86" exitCode=0 Jul 06 15:34:26.764712 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:26.764565 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerDied","Data":"2be61aca890bc463019913857b941eb819c8608a54205d79d4d4c082b5472f86"} Jul 06 15:34:27.600999 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:27.600969 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:27.601502 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:27.601062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:27.601502 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:27.601066 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jtj2c" podUID="2b0126b9-095c-4bbb-a0b6-a9c2e9359993" Jul 06 15:34:27.601502 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:27.601100 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:27.601502 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:27.601193 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2446q" podUID="a615b385-ad01-44ed-b97e-3bc14fd4e07b" Jul 06 15:34:27.601502 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:27.601257 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-45vqq" podUID="5d7e1678-72d8-4306-935b-a89536d24248" Jul 06 15:34:27.770179 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:27.770123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-46fm2" event={"ID":"a97b91af-ad9a-4b9a-abc4-6c6c56e85cde","Type":"ContainerStarted","Data":"31f9fe34fba21041e3dbf58f52b8823f2ffc653379bfc4d317eeb46ca145c628"} Jul 06 15:34:27.974751 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:27.974643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:28.056226 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.056186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:28.128714 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.128683 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:28.844726 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.844696 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-241.ec2.internal" event="NodeReady" Jul 06 15:34:28.845154 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.844818 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 15:34:28.900622 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.900569 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-46fm2" podStartSLOduration=9.310089916999999 podStartE2EDuration="36.900550792s" totalImagesPullingTime="27.590460875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:27.80936151 +0000 UTC m=+36.744111557" watchObservedRunningTime="2026-07-06 15:34:28.900550792 +0000 UTC m=+37.835300839" Jul 06 15:34:28.900968 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.900946 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dd9b99d4b-nlgf4"] Jul 06 15:34:28.915043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.915013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:28.918017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.917989 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 15:34:28.918017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.917991 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-98ctb\"" Jul 06 15:34:28.918218 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.918046 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 15:34:28.918218 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.918054 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 15:34:28.921763 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.921738 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd9b99d4b-nlgf4"] Jul 06 15:34:28.923675 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.923642 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 15:34:28.928249 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.928214 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4sbcx"] Jul 06 15:34:28.950305 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.949142 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mzkwj"] Jul 06 15:34:28.952024 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.951992 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:28.954895 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.954874 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 15:34:28.955047 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.954939 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8w9vb\"" Jul 06 15:34:28.955390 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.955369 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 15:34:28.971122 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.971091 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzkwj"] Jul 06 15:34:28.971122 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.971127 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4sbcx"] Jul 06 15:34:28.971306 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.971242 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:28.974444 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.974420 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 15:34:28.974780 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.974762 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 15:34:28.975041 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.975023 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pk7tw\"" Jul 06 15:34:28.975103 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:28.975078 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 15:34:29.009609 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt99f\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-kube-api-access-mt99f\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.009609 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-trusted-ca\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.009835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-tmp-dir\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.009835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009666 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-ca-trust-extracted\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.009835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-certificates\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.009835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-installation-pull-secrets\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.009835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.009835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-config-volume\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.009835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009834 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbgjg\" (UniqueName: \"kubernetes.io/projected/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-kube-api-access-kbgjg\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.010080 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-image-registry-private-configuration\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.010080 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.010080 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.009975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-bound-sa-token\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.110895 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.110859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-ca-trust-extracted\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.110895 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.110896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-certificates\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.110934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-installation-pull-secrets\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.110970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.110998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-config-volume\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.111126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbgjg\" (UniqueName: \"kubernetes.io/projected/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-kube-api-access-kbgjg\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.111126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-image-registry-private-configuration\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3352ee41-9ed0-4b96-942d-6f9e18f5c10b-cert\") pod \"ingress-canary-mzkwj\" (UID: \"3352ee41-9ed0-4b96-942d-6f9e18f5c10b\") " pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.111132 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.111156 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd9b99d4b-nlgf4: secret "image-registry-tls" not found Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.111251 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls podName:ff3ffc05-b70d-462c-82b4-dfdfbf102bff nodeName:}" failed. No retries permitted until 2026-07-06 15:34:29.611225887 +0000 UTC m=+38.545975929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls") pod "image-registry-dd9b99d4b-nlgf4" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff") : secret "image-registry-tls" not found Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj6bq\" (UniqueName: \"kubernetes.io/projected/3352ee41-9ed0-4b96-942d-6f9e18f5c10b-kube-api-access-fj6bq\") pod \"ingress-canary-mzkwj\" (UID: \"3352ee41-9ed0-4b96-942d-6f9e18f5c10b\") " pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111367 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-ca-trust-extracted\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111433 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-bound-sa-token\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.111456 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:34:29.111450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt99f\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-kube-api-access-mt99f\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111929 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111487 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-trusted-ca\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111929 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.111523 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls podName:2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:29.611499451 +0000 UTC m=+38.546249486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls") pod "dns-default-4sbcx" (UID: "2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52") : secret "dns-default-metrics-tls" not found Jul 06 15:34:29.111929 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-tmp-dir\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.111929 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111688 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-certificates\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.111929 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111708 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-config-volume\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.111929 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.111797 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-tmp-dir\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.112250 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.112232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-trusted-ca\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.115266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.115244 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-installation-pull-secrets\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.115357 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.115288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-image-registry-private-configuration\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.128599 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.128571 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbgjg\" (UniqueName: \"kubernetes.io/projected/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-kube-api-access-kbgjg\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.128720 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.128688 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-bound-sa-token\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.128810 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.128795 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt99f\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-kube-api-access-mt99f\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.212738 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.212645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3352ee41-9ed0-4b96-942d-6f9e18f5c10b-cert\") pod \"ingress-canary-mzkwj\" (UID: \"3352ee41-9ed0-4b96-942d-6f9e18f5c10b\") " pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:29.212738 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.212684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj6bq\" (UniqueName: \"kubernetes.io/projected/3352ee41-9ed0-4b96-942d-6f9e18f5c10b-kube-api-access-fj6bq\") pod \"ingress-canary-mzkwj\" (UID: \"3352ee41-9ed0-4b96-942d-6f9e18f5c10b\") " pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:29.215327 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.215302 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3352ee41-9ed0-4b96-942d-6f9e18f5c10b-cert\") pod \"ingress-canary-mzkwj\" (UID: \"3352ee41-9ed0-4b96-942d-6f9e18f5c10b\") " pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:29.221576 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.221552 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj6bq\" (UniqueName: \"kubernetes.io/projected/3352ee41-9ed0-4b96-942d-6f9e18f5c10b-kube-api-access-fj6bq\") pod \"ingress-canary-mzkwj\" (UID: \"3352ee41-9ed0-4b96-942d-6f9e18f5c10b\") " pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:29.279783 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.279737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzkwj" Jul 06 15:34:29.484871 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.484629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzkwj"] Jul 06 15:34:29.487749 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:29.487721 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3352ee41_9ed0_4b96_942d_6f9e18f5c10b.slice/crio-d5b5205a052e3777d8696c7a042d7f12e62bdada263c50e03ae0e9625234c14d WatchSource:0}: Error finding container d5b5205a052e3777d8696c7a042d7f12e62bdada263c50e03ae0e9625234c14d: Status 404 returned error can't find the container with id d5b5205a052e3777d8696c7a042d7f12e62bdada263c50e03ae0e9625234c14d Jul 06 15:34:29.597592 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.597549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:29.597780 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.597550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:29.597780 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.597685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:29.601557 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.601539 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 15:34:29.601697 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.601542 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 15:34:29.601697 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.601660 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 15:34:29.601813 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.601742 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v99lt\"" Jul 06 15:34:29.601868 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.601853 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w485s\"" Jul 06 15:34:29.601917 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.601892 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 15:34:29.616123 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.616101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:29.616249 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.616193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:29.616325 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.616290 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 15:34:29.616325 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.616308 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd9b99d4b-nlgf4: secret "image-registry-tls" not found Jul 06 15:34:29.616413 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.616326 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:34:29.616413 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.616365 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls podName:ff3ffc05-b70d-462c-82b4-dfdfbf102bff nodeName:}" failed. No retries permitted until 2026-07-06 15:34:30.616345754 +0000 UTC m=+39.551095782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls") pod "image-registry-dd9b99d4b-nlgf4" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff") : secret "image-registry-tls" not found Jul 06 15:34:29.616413 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:29.616385 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls podName:2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:30.616374429 +0000 UTC m=+39.551124455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls") pod "dns-default-4sbcx" (UID: "2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52") : secret "dns-default-metrics-tls" not found Jul 06 15:34:29.776405 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:29.776312 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzkwj" event={"ID":"3352ee41-9ed0-4b96-942d-6f9e18f5c10b","Type":"ContainerStarted","Data":"d5b5205a052e3777d8696c7a042d7f12e62bdada263c50e03ae0e9625234c14d"} Jul 06 15:34:30.625183 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:30.625133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:30.625645 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:30.625229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:30.625645 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:30.625308 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:34:30.625645 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:30.625357 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 15:34:30.625645 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:30.625370 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd9b99d4b-nlgf4: secret "image-registry-tls" not found Jul 06 15:34:30.625645 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:30.625383 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls podName:2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:32.625366843 +0000 UTC m=+41.560116867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls") pod "dns-default-4sbcx" (UID: "2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52") : secret "dns-default-metrics-tls" not found Jul 06 15:34:30.625645 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:30.625408 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls podName:ff3ffc05-b70d-462c-82b4-dfdfbf102bff nodeName:}" failed. No retries permitted until 2026-07-06 15:34:32.625396863 +0000 UTC m=+41.560146888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls") pod "image-registry-dd9b99d4b-nlgf4" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff") : secret "image-registry-tls" not found Jul 06 15:34:31.279034 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:31.278999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:31.345733 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:31.345698 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:32.174148 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.174115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:32.317800 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.317768 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:32.400388 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.400300 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:32.500434 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.500403 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:32.611468 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.611436 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:32.638890 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.638856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:32.639010 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.638923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:32.639047 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:32.639009 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 15:34:32.639047 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:32.639024 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd9b99d4b-nlgf4: secret "image-registry-tls" not found Jul 06 15:34:32.639047 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:32.639024 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:34:32.639381 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:32.639086 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls podName:2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:36.639070934 +0000 UTC m=+45.573820961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls") pod "dns-default-4sbcx" (UID: "2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52") : secret "dns-default-metrics-tls" not found Jul 06 15:34:32.639381 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:32.639101 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls podName:ff3ffc05-b70d-462c-82b4-dfdfbf102bff nodeName:}" failed. No retries permitted until 2026-07-06 15:34:36.639095333 +0000 UTC m=+45.573845358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls") pod "image-registry-dd9b99d4b-nlgf4" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff") : secret "image-registry-tls" not found Jul 06 15:34:32.710490 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.710398 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:32.785675 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.785626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzkwj" event={"ID":"3352ee41-9ed0-4b96-942d-6f9e18f5c10b","Type":"ContainerStarted","Data":"50d8128b1f083fddae286fee0ca26cf4277b087bab47aad8289f4a0d57ccc29e"} Jul 06 15:34:32.804866 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.804816 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mzkwj" podStartSLOduration=2.202398001 podStartE2EDuration="4.804801308s" totalImagesPullingTime="2.602403307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:32.804453353 +0000 UTC m=+41.739203400" watchObservedRunningTime="2026-07-06 15:34:32.804801308 +0000 UTC m=+41.739551332" Jul 06 15:34:32.996714 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.996647 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s7d97"] Jul 06 15:34:32.998531 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:32.998516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.000919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.000893 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 15:34:33.001039 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.000925 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rtrq\"" Jul 06 15:34:33.001039 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.000958 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 15:34:33.001039 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.000899 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 15:34:33.001244 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.001230 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 15:34:33.008857 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.008833 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:33.012291 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.012269 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s7d97"] Jul 06 15:34:33.076939 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.076900 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:33.142359 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.142321 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.142530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.142365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.142530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.142391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9cw\" (UniqueName: \"kubernetes.io/projected/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-api-access-xb9cw\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.142530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.142486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-crio-socket\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.142530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.142513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-data-volume\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.243290 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.243251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.243302 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.243325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9cw\" (UniqueName: \"kubernetes.io/projected/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-api-access-xb9cw\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.243391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-crio-socket\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.243397 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.243416 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-data-volume\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.243482 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:33.743461631 +0000 UTC m=+42.678211656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : configmap "kube-rbac-proxy" not found Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.243403 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.243570 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:33.743552708 +0000 UTC m=+42.678302740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : secret "insights-runtime-extractor-tls" not found Jul 06 15:34:33.243776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.243590 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-crio-socket\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.244088 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.243800 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-data-volume\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.256154 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.255996 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9cw\" (UniqueName: \"kubernetes.io/projected/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-api-access-xb9cw\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.375672 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.375637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:33.475848 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.475815 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:33.746620 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.746567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.746620 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:33.746628 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:33.746935 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.746725 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:34:33.746935 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.746799 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:34.746782873 +0000 UTC m=+43.681532899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : secret "insights-runtime-extractor-tls" not found Jul 06 15:34:33.746935 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.746726 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:34:33.746935 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:33.746880 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:34.746864166 +0000 UTC m=+43.681614193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : configmap "kube-rbac-proxy" not found Jul 06 15:34:34.031157 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.031125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:34.100863 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.100840 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:34.167506 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.167480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:34.755333 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.755281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:34.755333 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.755335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:34.755773 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:34.755430 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:34:34.755773 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:34.755447 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:34:34.755773 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:34.755483 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:36.75546807 +0000 UTC m=+45.690218096 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : configmap "kube-rbac-proxy" not found Jul 06 15:34:34.755773 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:34.755504 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:36.755489311 +0000 UTC m=+45.690239339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : secret "insights-runtime-extractor-tls" not found Jul 06 15:34:34.950191 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.950140 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-v5s5g"] Jul 06 15:34:34.955226 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.955210 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:34.959076 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.959054 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 15:34:34.959227 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.959142 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 15:34:34.960071 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.960049 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 15:34:34.960152 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.960052 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 15:34:34.960152 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.960105 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-qntmz\"" Jul 06 15:34:34.965990 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:34.965971 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-v5s5g"] Jul 06 15:34:35.008838 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.008762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:35.057227 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.057190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c5c33281-c731-4f1b-aa20-9abea97a9d9b-signing-key\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.057412 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.057235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c5c33281-c731-4f1b-aa20-9abea97a9d9b-signing-cabundle\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.057412 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.057255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xjx\" (UniqueName: \"kubernetes.io/projected/c5c33281-c731-4f1b-aa20-9abea97a9d9b-kube-api-access-n7xjx\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.121985 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.121944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:35.158098 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.158063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c5c33281-c731-4f1b-aa20-9abea97a9d9b-signing-key\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.158226 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.158110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c5c33281-c731-4f1b-aa20-9abea97a9d9b-signing-cabundle\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.158226 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.158126 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xjx\" (UniqueName: \"kubernetes.io/projected/c5c33281-c731-4f1b-aa20-9abea97a9d9b-kube-api-access-n7xjx\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.160824 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.160802 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c5c33281-c731-4f1b-aa20-9abea97a9d9b-signing-key\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.168101 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.168078 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xjx\" (UniqueName: \"kubernetes.io/projected/c5c33281-c731-4f1b-aa20-9abea97a9d9b-kube-api-access-n7xjx\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.169278 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.169260 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c5c33281-c731-4f1b-aa20-9abea97a9d9b-signing-cabundle\") pod \"service-ca-758f599c5-v5s5g\" (UID: \"c5c33281-c731-4f1b-aa20-9abea97a9d9b\") " pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.264238 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.264113 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-v5s5g" Jul 06 15:34:35.439548 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.439515 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-v5s5g"] Jul 06 15:34:35.443676 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:35.443646 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c33281_c731_4f1b_aa20_9abea97a9d9b.slice/crio-74d02d85971c2774803fdfee5c070b9eb8ea4bb59cdb87e1d0cddaab85286a47 WatchSource:0}: Error finding container 74d02d85971c2774803fdfee5c070b9eb8ea4bb59cdb87e1d0cddaab85286a47: Status 404 returned error can't find the container with id 74d02d85971c2774803fdfee5c070b9eb8ea4bb59cdb87e1d0cddaab85286a47 Jul 06 15:34:35.793937 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:35.793897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-v5s5g" event={"ID":"c5c33281-c731-4f1b-aa20-9abea97a9d9b","Type":"ContainerStarted","Data":"74d02d85971c2774803fdfee5c070b9eb8ea4bb59cdb87e1d0cddaab85286a47"} Jul 06 15:34:36.083408 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.083322 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spgk5_fbcf2182-b539-45cd-bc1b-9fa31f2f5b88/dns-node-resolver/0.log" Jul 06 15:34:36.083573 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.083549 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:36.215710 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.215670 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:36.447877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.447751 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:36.670879 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.670843 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:36.671046 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.670925 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:36.671046 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.671016 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 15:34:36.671046 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.671032 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-dd9b99d4b-nlgf4: secret "image-registry-tls" not found Jul 06 15:34:36.671419 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.671086 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls podName:ff3ffc05-b70d-462c-82b4-dfdfbf102bff nodeName:}" failed. No retries permitted until 2026-07-06 15:34:44.671071518 +0000 UTC m=+53.605821544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls") pod "image-registry-dd9b99d4b-nlgf4" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff") : secret "image-registry-tls" not found Jul 06 15:34:36.671419 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.671089 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 15:34:36.671830 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.671804 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls podName:2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52 nodeName:}" failed. No retries permitted until 2026-07-06 15:34:44.671123703 +0000 UTC m=+53.605873732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls") pod "dns-default-4sbcx" (UID: "2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52") : secret "dns-default-metrics-tls" not found Jul 06 15:34:36.672029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.672011 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9xfl_c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16/node-ca/0.log" Jul 06 15:34:36.771571 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.771486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:36.771733 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.771618 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:34:36.771733 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:36.771637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:36.771733 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.771695 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:40.771671582 +0000 UTC m=+49.706421619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : configmap "kube-rbac-proxy" not found Jul 06 15:34:36.771914 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.771736 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:34:36.771914 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:36.771793 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:40.771777793 +0000 UTC m=+49.706527825 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : secret "insights-runtime-extractor-tls" not found Jul 06 15:34:37.666317 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:37.666291 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mzkwj_3352ee41-9ed0-4b96-942d-6f9e18f5c10b/serve-healthcheck-canary/0.log" Jul 06 15:34:38.699264 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:38.698986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:38.765928 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:38.765875 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:38.802611 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:38.802580 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-v5s5g" event={"ID":"c5c33281-c731-4f1b-aa20-9abea97a9d9b","Type":"ContainerStarted","Data":"f5f95b3cf72bb42975cbd73c74eb084e417f9507054823cc8839e173fb0b30c8"} Jul 06 15:34:38.822825 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:38.822776 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-v5s5g" podStartSLOduration=2.01326145 podStartE2EDuration="4.822760744s" totalImagesPullingTime="2.809499294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:38.822416068 +0000 UTC m=+47.757166119" watchObservedRunningTime="2026-07-06 15:34:38.822760744 +0000 UTC m=+47.757510790" Jul 06 15:34:39.329100 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:39.329064 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:39.427657 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:39.427623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:39.551716 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:39.551681 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:39.681841 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:39.681752 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:39.828440 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:39.828405 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:40.129751 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:40.129718 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:40.331871 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:40.331830 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:40.805152 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:40.805119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:40.805349 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:40.805157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:40.805349 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:40.805292 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:34:40.805349 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:40.805329 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 15:34:40.805495 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:40.805367 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:48.805350515 +0000 UTC m=+57.740100540 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : secret "insights-runtime-extractor-tls" not found Jul 06 15:34:40.805495 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:40.805387 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:34:48.80537507 +0000 UTC m=+57.740125094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : configmap "kube-rbac-proxy" not found Jul 06 15:34:41.126314 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:41.126232 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:41.339735 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:41.339701 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:41.699372 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:41.699336 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:41.910114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:41.910081 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:42.494116 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:42.494075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:42.851676 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:42.851641 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:42.926292 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:42.926252 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:43.092238 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:43.092209 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:43.288558 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:43.288528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:43.496119 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:43.496078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:43.596537 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:43.596448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:43.697765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:43.697734 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:44.707552 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.707524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:44.736067 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.736035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:44.736226 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.736098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:44.739515 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.739485 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52-metrics-tls\") pod \"dns-default-4sbcx\" (UID: \"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52\") " pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:44.739630 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.739493 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"image-registry-dd9b99d4b-nlgf4\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:44.826302 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.826267 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:44.854412 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.854378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:44.860570 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.860539 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:44.978142 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:44.978109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:45.014652 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.014553 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dd9b99d4b-nlgf4"] Jul 06 15:34:45.017143 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:45.017115 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff3ffc05_b70d_462c_82b4_dfdfbf102bff.slice/crio-218f63ebef01d1a0679fa48a92df6aba5437bf7a5716108a77bc4574a2e94707 WatchSource:0}: Error finding container 218f63ebef01d1a0679fa48a92df6aba5437bf7a5716108a77bc4574a2e94707: Status 404 returned error can't find the container with id 218f63ebef01d1a0679fa48a92df6aba5437bf7a5716108a77bc4574a2e94707 Jul 06 15:34:45.020562 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.020537 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4sbcx"] Jul 06 15:34:45.023883 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:45.023861 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec270e9_050d_4d6e_b2aa_9f6b7b4b2f52.slice/crio-2706ea031c1556e8a046d2bc81060ec08088982dd8646a298124d246c0f590ae WatchSource:0}: Error finding container 2706ea031c1556e8a046d2bc81060ec08088982dd8646a298124d246c0f590ae: Status 404 returned error can't find the container with id 2706ea031c1556e8a046d2bc81060ec08088982dd8646a298124d246c0f590ae Jul 06 15:34:45.132261 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.132226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:45.340087 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.340054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:45.446927 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.446899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:45.548374 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.548338 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:45.704138 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.704051 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:45.824629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.824587 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" event={"ID":"ff3ffc05-b70d-462c-82b4-dfdfbf102bff","Type":"ContainerStarted","Data":"18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0"} Jul 06 15:34:45.824629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.824637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" event={"ID":"ff3ffc05-b70d-462c-82b4-dfdfbf102bff","Type":"ContainerStarted","Data":"218f63ebef01d1a0679fa48a92df6aba5437bf7a5716108a77bc4574a2e94707"} Jul 06 15:34:45.825130 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.824714 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:34:45.825887 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.825868 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sbcx" event={"ID":"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52","Type":"ContainerStarted","Data":"2706ea031c1556e8a046d2bc81060ec08088982dd8646a298124d246c0f590ae"} Jul 06 15:34:45.846757 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:45.846701 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" podStartSLOduration=34.846686219 podStartE2EDuration="34.846686219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:45.846135693 +0000 UTC m=+54.780885759" watchObservedRunningTime="2026-07-06 15:34:45.846686219 +0000 UTC m=+54.781436266" Jul 06 15:34:46.691526 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:46.691495 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:47.123837 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:47.123800 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:47.329082 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:47.329046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:47.500464 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:47.500298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:47.687319 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:47.687270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:48.694498 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.694463 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:48.835651 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.835610 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sbcx" event={"ID":"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52","Type":"ContainerStarted","Data":"fada50b747ebbfa37766ae78108a3649a6104f591d7ec668aa145cc9ae81b8a6"} Jul 06 15:34:48.835818 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.835665 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4sbcx" event={"ID":"2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52","Type":"ContainerStarted","Data":"763fe62c24255965afb3451fa80600e76816b840aae3a03d7236367b0c8966ae"} Jul 06 15:34:48.835818 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.835706 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4sbcx" Jul 06 15:34:48.872313 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.872277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:48.872480 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.872324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:48.872480 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:48.872426 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 15:34:48.872571 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:34:48.872491 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls podName:b7c5c181-6f88-4d46-b7d9-bdf86ca0acef nodeName:}" failed. No retries permitted until 2026-07-06 15:35:04.872474866 +0000 UTC m=+73.807224902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s7d97" (UID: "b7c5c181-6f88-4d46-b7d9-bdf86ca0acef") : secret "insights-runtime-extractor-tls" not found Jul 06 15:34:48.872827 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.872808 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:34:48.880635 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:48.880593 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4sbcx" podStartSLOduration=17.779740263 podStartE2EDuration="20.880579752s" totalImagesPullingTime="3.100839489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:34:48.878940902 +0000 UTC m=+57.813690971" watchObservedRunningTime="2026-07-06 15:34:48.880579752 +0000 UTC m=+57.815329799" Jul 06 15:34:49.099990 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:49.099957 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:49.291227 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:49.291187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:49.519638 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:49.519559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:49.703529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:49.703496 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:50.092540 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:50.092510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:51.055021 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:51.054990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:51.172241 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:51.172212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:54.781442 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:54.781402 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qlr26" Jul 06 15:34:55.615036 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.615006 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk"] Jul 06 15:34:55.646344 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.646311 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52"] Jul 06 15:34:55.646496 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.646437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.650842 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.650788 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 15:34:55.651062 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.651044 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 15:34:55.651150 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.651102 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 15:34:55.651505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.651487 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 15:34:55.666281 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.666254 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:55.669076 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.669055 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk"] Jul 06 15:34:55.670112 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.670082 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 06 15:34:55.670289 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.670270 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-26vnj\"" Jul 06 15:34:55.686394 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.686364 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52"] Jul 06 15:34:55.700986 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.700956 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dd9b99d4b-nlgf4"] Jul 06 15:34:55.724913 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.724881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b4292c-2929-456e-a4e4-607e4ac8e703-tmp\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.725086 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.724943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a1b4292c-2929-456e-a4e4-607e4ac8e703-klusterlet-config\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.725086 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.724981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sp6k\" (UniqueName: \"kubernetes.io/projected/cd1161c6-a49f-4994-950f-0082269811a8-kube-api-access-7sp6k\") pod \"managed-serviceaccount-addon-agent-7c465b574d-bnj52\" (UID: \"cd1161c6-a49f-4994-950f-0082269811a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:55.725086 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.725017 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f49pb\" (UniqueName: \"kubernetes.io/projected/a1b4292c-2929-456e-a4e4-607e4ac8e703-kube-api-access-f49pb\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.725214 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.725109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd1161c6-a49f-4994-950f-0082269811a8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c465b574d-bnj52\" (UID: \"cd1161c6-a49f-4994-950f-0082269811a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:55.825490 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.825455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b4292c-2929-456e-a4e4-607e4ac8e703-tmp\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.825885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.825512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a1b4292c-2929-456e-a4e4-607e4ac8e703-klusterlet-config\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.825885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.825631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sp6k\" (UniqueName: \"kubernetes.io/projected/cd1161c6-a49f-4994-950f-0082269811a8-kube-api-access-7sp6k\") pod \"managed-serviceaccount-addon-agent-7c465b574d-bnj52\" (UID: \"cd1161c6-a49f-4994-950f-0082269811a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:55.825885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.825665 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f49pb\" (UniqueName: \"kubernetes.io/projected/a1b4292c-2929-456e-a4e4-607e4ac8e703-kube-api-access-f49pb\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.825885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.825738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd1161c6-a49f-4994-950f-0082269811a8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c465b574d-bnj52\" (UID: \"cd1161c6-a49f-4994-950f-0082269811a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:55.826096 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.825880 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a1b4292c-2929-456e-a4e4-607e4ac8e703-tmp\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.828307 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.828278 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cd1161c6-a49f-4994-950f-0082269811a8-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c465b574d-bnj52\" (UID: \"cd1161c6-a49f-4994-950f-0082269811a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:55.828480 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.828458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a1b4292c-2929-456e-a4e4-607e4ac8e703-klusterlet-config\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.847125 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.847096 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f49pb\" (UniqueName: \"kubernetes.io/projected/a1b4292c-2929-456e-a4e4-607e4ac8e703-kube-api-access-f49pb\") pod \"klusterlet-addon-workmgr-7596479755-kj7gk\" (UID: \"a1b4292c-2929-456e-a4e4-607e4ac8e703\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.847633 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.847615 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sp6k\" (UniqueName: \"kubernetes.io/projected/cd1161c6-a49f-4994-950f-0082269811a8-kube-api-access-7sp6k\") pod \"managed-serviceaccount-addon-agent-7c465b574d-bnj52\" (UID: \"cd1161c6-a49f-4994-950f-0082269811a8\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:55.849358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.849332 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:55.955382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.955288 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:34:55.982305 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:55.982280 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" Jul 06 15:34:56.063965 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:56.063834 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:56.121903 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:56.121875 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk"] Jul 06 15:34:56.124879 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:56.124851 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b4292c_2929_456e_a4e4_607e4ac8e703.slice/crio-62fe20ffd696dd60da7a61a6aca52ca7f04e170d2d6c96092a3a5a694aa16012 WatchSource:0}: Error finding container 62fe20ffd696dd60da7a61a6aca52ca7f04e170d2d6c96092a3a5a694aa16012: Status 404 returned error can't find the container with id 62fe20ffd696dd60da7a61a6aca52ca7f04e170d2d6c96092a3a5a694aa16012 Jul 06 15:34:56.155395 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:56.155366 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52"] Jul 06 15:34:56.158348 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:56.158321 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd1161c6_a49f_4994_950f_0082269811a8.slice/crio-7a1bfe7457dc394971c20481197c6c978f3bff57932a32aa4d2c1f5c968d8dde WatchSource:0}: Error finding container 7a1bfe7457dc394971c20481197c6c978f3bff57932a32aa4d2c1f5c968d8dde: Status 404 returned error can't find the container with id 7a1bfe7457dc394971c20481197c6c978f3bff57932a32aa4d2c1f5c968d8dde Jul 06 15:34:56.241231 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:56.241125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:56.398477 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:56.398443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:34:56.856499 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:56.856457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" event={"ID":"a1b4292c-2929-456e-a4e4-607e4ac8e703","Type":"ContainerStarted","Data":"62fe20ffd696dd60da7a61a6aca52ca7f04e170d2d6c96092a3a5a694aa16012"} Jul 06 15:34:56.859400 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:56.859365 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" event={"ID":"cd1161c6-a49f-4994-950f-0082269811a8","Type":"ContainerStarted","Data":"7a1bfe7457dc394971c20481197c6c978f3bff57932a32aa4d2c1f5c968d8dde"} Jul 06 15:34:57.336591 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.336036 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:57.336591 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.336152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:57.339208 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.339155 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 15:34:57.339678 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.339468 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 15:34:57.350556 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.350525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d7e1678-72d8-4306-935b-a89536d24248-metrics-certs\") pod \"network-metrics-daemon-45vqq\" (UID: \"5d7e1678-72d8-4306-935b-a89536d24248\") " pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:57.354343 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.354288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2b0126b9-095c-4bbb-a0b6-a9c2e9359993-original-pull-secret\") pod \"global-pull-secret-syncer-jtj2c\" (UID: \"2b0126b9-095c-4bbb-a0b6-a9c2e9359993\") " pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:57.436993 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.436953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:57.440506 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.440474 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 15:34:57.452183 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.452113 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 15:34:57.471803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.471763 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjg22\" (UniqueName: \"kubernetes.io/projected/a615b385-ad01-44ed-b97e-3bc14fd4e07b-kube-api-access-jjg22\") pod \"network-check-target-2446q\" (UID: \"a615b385-ad01-44ed-b97e-3bc14fd4e07b\") " pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:57.510762 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.510717 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-w485s\"" Jul 06 15:34:57.515466 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.515438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v99lt\"" Jul 06 15:34:57.517631 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.517611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:34:57.517942 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.517917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jtj2c" Jul 06 15:34:57.523077 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.523049 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-45vqq" Jul 06 15:34:57.726579 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.726543 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2446q"] Jul 06 15:34:57.731770 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:57.731737 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda615b385_ad01_44ed_b97e_3bc14fd4e07b.slice/crio-78d7cfdbcac0e1b4070d1161b83af5548b72aea3e7a4d9765154651e498e6fda WatchSource:0}: Error finding container 78d7cfdbcac0e1b4070d1161b83af5548b72aea3e7a4d9765154651e498e6fda: Status 404 returned error can't find the container with id 78d7cfdbcac0e1b4070d1161b83af5548b72aea3e7a4d9765154651e498e6fda Jul 06 15:34:57.766797 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.766718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-45vqq"] Jul 06 15:34:57.771050 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:57.771011 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7e1678_72d8_4306_935b_a89536d24248.slice/crio-1df539b618809aae5c44c0f263e5d65a64205c743db0e56fec2635d3b0bbb6fa WatchSource:0}: Error finding container 1df539b618809aae5c44c0f263e5d65a64205c743db0e56fec2635d3b0bbb6fa: Status 404 returned error can't find the container with id 1df539b618809aae5c44c0f263e5d65a64205c743db0e56fec2635d3b0bbb6fa Jul 06 15:34:57.786954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.786890 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jtj2c"] Jul 06 15:34:57.792208 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:34:57.792176 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0126b9_095c_4bbb_a0b6_a9c2e9359993.slice/crio-e10f45cd2b6bfd74aa7d65396c79f22a7b2142ca22f243036adf945143ec4bd2 WatchSource:0}: Error finding container e10f45cd2b6bfd74aa7d65396c79f22a7b2142ca22f243036adf945143ec4bd2: Status 404 returned error can't find the container with id e10f45cd2b6bfd74aa7d65396c79f22a7b2142ca22f243036adf945143ec4bd2 Jul 06 15:34:57.863910 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.863817 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-45vqq" event={"ID":"5d7e1678-72d8-4306-935b-a89536d24248","Type":"ContainerStarted","Data":"1df539b618809aae5c44c0f263e5d65a64205c743db0e56fec2635d3b0bbb6fa"} Jul 06 15:34:57.865942 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.865807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jtj2c" event={"ID":"2b0126b9-095c-4bbb-a0b6-a9c2e9359993","Type":"ContainerStarted","Data":"e10f45cd2b6bfd74aa7d65396c79f22a7b2142ca22f243036adf945143ec4bd2"} Jul 06 15:34:57.867772 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:57.867740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2446q" event={"ID":"a615b385-ad01-44ed-b97e-3bc14fd4e07b","Type":"ContainerStarted","Data":"78d7cfdbcac0e1b4070d1161b83af5548b72aea3e7a4d9765154651e498e6fda"} Jul 06 15:34:58.840755 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:34:58.840718 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4sbcx" Jul 06 15:35:00.020474 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.020431 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:00.116264 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.116184 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:00.232406 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.232332 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:00.306245 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.306140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:00.579398 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.579317 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:00.682423 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.682386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:00.785068 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.785036 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:00.931053 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:00.931022 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:02.126231 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:02.126199 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:02.225589 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:02.225557 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:03.159223 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.159185 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:03.272347 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.272308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:03.888823 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.888783 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" event={"ID":"a1b4292c-2929-456e-a4e4-607e4ac8e703","Type":"ContainerStarted","Data":"a4bdbcc59ccb27b74ff9c968949d5710c28f6e58ef9240011f850aefd28840a0"} Jul 06 15:35:03.889017 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.888949 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:35:03.890809 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.890781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-45vqq" event={"ID":"5d7e1678-72d8-4306-935b-a89536d24248","Type":"ContainerStarted","Data":"87361db6a5ddf841b0666971e008590b0c42f03285d9c168e67029e81ff79b29"} Jul 06 15:35:03.890913 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.890826 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-45vqq" event={"ID":"5d7e1678-72d8-4306-935b-a89536d24248","Type":"ContainerStarted","Data":"1aeb67e2b8615354ce56dddc8ba606cd4c47313640ea5af61fba143b3cbd9738"} Jul 06 15:35:03.890969 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.890915 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" Jul 06 15:35:03.892386 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.892356 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" event={"ID":"cd1161c6-a49f-4994-950f-0082269811a8","Type":"ContainerStarted","Data":"a3cb38df87d169f379f4a9339fb4e54494a99753e19149bd7ecf83a7a3029cbe"} Jul 06 15:35:03.893648 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.893629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jtj2c" event={"ID":"2b0126b9-095c-4bbb-a0b6-a9c2e9359993","Type":"ContainerStarted","Data":"16dce4efa0f8cd6825e7d879606a5fc45c9318c9810f3154c886a530cb914301"} Jul 06 15:35:03.895050 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.895032 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2446q" event={"ID":"a615b385-ad01-44ed-b97e-3bc14fd4e07b","Type":"ContainerStarted","Data":"d7bb8818098fe2264eeacf9601f672fdb006d90e468bc7bcccac08c2457f0061"} Jul 06 15:35:03.895315 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.895157 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:35:03.952401 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.952343 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7596479755-kj7gk" podStartSLOduration=2.947996119 podStartE2EDuration="8.952324328s" totalImagesPullingTime="6.004328209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:03.921064565 +0000 UTC m=+72.855814613" watchObservedRunningTime="2026-07-06 15:35:03.952324328 +0000 UTC m=+72.887074375" Jul 06 15:35:03.952569 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:03.952543 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2446q" podStartSLOduration=67.110651652 podStartE2EDuration="1m12.952538205s" totalImagesPullingTime="5.841886553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:03.952309533 +0000 UTC m=+72.887059579" watchObservedRunningTime="2026-07-06 15:35:03.952538205 +0000 UTC m=+72.887288252" Jul 06 15:35:04.047779 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.047676 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jtj2c" podStartSLOduration=66.254965739 podStartE2EDuration="1m12.047656169s" totalImagesPullingTime="5.79269043s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:04.046528907 +0000 UTC m=+72.981278951" watchObservedRunningTime="2026-07-06 15:35:04.047656169 +0000 UTC m=+72.982406216" Jul 06 15:35:04.096976 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.096914 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-45vqq" podStartSLOduration=66.301194264 podStartE2EDuration="1m12.096898114s" totalImagesPullingTime="5.79570385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:04.093496821 +0000 UTC m=+73.028246865" watchObservedRunningTime="2026-07-06 15:35:04.096898114 +0000 UTC m=+73.031648160" Jul 06 15:35:04.143721 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.143667 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c465b574d-bnj52" podStartSLOduration=3.172690774 podStartE2EDuration="9.143649727s" totalImagesPullingTime="5.970958953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:04.143184019 +0000 UTC m=+73.077934068" watchObservedRunningTime="2026-07-06 15:35:04.143649727 +0000 UTC m=+73.078399773" Jul 06 15:35:04.291588 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.291553 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:04.401569 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.401487 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:04.507500 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.507465 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:04.896185 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.896124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:35:04.899973 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:04.899939 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b7c5c181-6f88-4d46-b7d9-bdf86ca0acef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s7d97\" (UID: \"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef\") " pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:35:05.110309 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:05.110269 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5rtrq\"" Jul 06 15:35:05.118417 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:05.118389 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s7d97" Jul 06 15:35:05.252955 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:05.250381 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s7d97"] Jul 06 15:35:05.253833 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:35:05.253755 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c5c181_6f88_4d46_b7d9_bdf86ca0acef.slice/crio-4ada0de4444a8b706633dbf6c1f55f1cd68378d79c58866aa423f715931d8494 WatchSource:0}: Error finding container 4ada0de4444a8b706633dbf6c1f55f1cd68378d79c58866aa423f715931d8494: Status 404 returned error can't find the container with id 4ada0de4444a8b706633dbf6c1f55f1cd68378d79c58866aa423f715931d8494 Jul 06 15:35:05.710091 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:05.710045 2575 patch_prober.go:28] interesting pod/image-registry-dd9b99d4b-nlgf4 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 15:35:05.710557 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:05.710119 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" podUID="ff3ffc05-b70d-462c-82b4-dfdfbf102bff" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 15:35:05.909064 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:05.909015 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s7d97" event={"ID":"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef","Type":"ContainerStarted","Data":"4ada0de4444a8b706633dbf6c1f55f1cd68378d79c58866aa423f715931d8494"} Jul 06 15:35:07.085538 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:07.085493 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:08.362692 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:08.362643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:08.506143 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:08.506104 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:09.581998 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:09.581960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:09.906308 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:09.906221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:09.925685 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:09.925651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s7d97" event={"ID":"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef","Type":"ContainerStarted","Data":"9674bfa96007e33300b9989574203dfbfde05ae96c7dd7563e5e532e24e690d3"} Jul 06 15:35:09.925840 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:09.925705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s7d97" event={"ID":"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef","Type":"ContainerStarted","Data":"ac95370bebaddd19e2de2f08b61ace40b042a3b0dd15b18cad99f97a1be87dc9"} Jul 06 15:35:12.938875 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:12.938840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s7d97" event={"ID":"b7c5c181-6f88-4d46-b7d9-bdf86ca0acef","Type":"ContainerStarted","Data":"d7052456d90d131e4c0fb083071dde32bd976b8df326b1268ae0e2867eb0b501"} Jul 06 15:35:12.961105 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:12.961047 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s7d97" podStartSLOduration=34.2842894 podStartE2EDuration="40.961026028s" totalImagesPullingTime="6.676736628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:34:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:12.96083031 +0000 UTC m=+81.895580370" watchObservedRunningTime="2026-07-06 15:35:12.961026028 +0000 UTC m=+81.895776076" Jul 06 15:35:12.965912 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:12.965886 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:13.111918 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:13.111887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.183434 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.183406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.248029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.247990 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9"] Jul 06 15:35:14.252311 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.252289 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.257250 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.257224 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 06 15:35:14.258094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.258057 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 15:35:14.258094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.258077 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-fn7ld\"" Jul 06 15:35:14.258819 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.258636 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 06 15:35:14.258819 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.258644 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 15:35:14.258819 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.258721 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 15:35:14.266459 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.266434 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9"] Jul 06 15:35:14.295587 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.295550 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-564pg"] Jul 06 15:35:14.299506 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.299489 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.302103 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.302079 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 15:35:14.302255 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.302220 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 15:35:14.302458 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.302438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7t9r2\"" Jul 06 15:35:14.302975 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.302957 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 15:35:14.306134 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.306109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.369682 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-tls\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.369682 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-textfile\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.369914 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f63d001-e301-4d65-b409-cda9932c4ea3-metrics-client-ca\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.369914 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.369914 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-wtmp\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.369914 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369861 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hf4j\" (UniqueName: \"kubernetes.io/projected/0f63d001-e301-4d65-b409-cda9932c4ea3-kube-api-access-6hf4j\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.370094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-sys\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.370094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.370094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9jxg\" (UniqueName: \"kubernetes.io/projected/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-kube-api-access-f9jxg\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.370094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.369988 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.370094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.370030 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.370094 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.370070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-accelerators-collector-config\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.370336 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.370114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-root\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.401064 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.401033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.471565 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-wtmp\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.471565 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hf4j\" (UniqueName: \"kubernetes.io/projected/0f63d001-e301-4d65-b409-cda9932c4ea3-kube-api-access-6hf4j\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.471565 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-sys\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.471808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.471808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-sys\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.471808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471667 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-wtmp\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.471808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9jxg\" (UniqueName: \"kubernetes.io/projected/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-kube-api-access-f9jxg\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.471808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.471808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.471808 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-accelerators-collector-config\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:35:14.471845 2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:35:14.471922 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-tls podName:7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb nodeName:}" failed. No retries permitted until 2026-07-06 15:35:14.971899541 +0000 UTC m=+83.906649569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-66zd9" (UID: "7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb") : secret "openshift-state-metrics-tls" not found Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-root\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.471986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-tls\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-textfile\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472017 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f63d001-e301-4d65-b409-cda9932c4ea3-root\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f63d001-e301-4d65-b409-cda9932c4ea3-metrics-client-ca\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472127 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472555 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-textfile\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472555 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472417 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-accelerators-collector-config\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472657 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472559 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f63d001-e301-4d65-b409-cda9932c4ea3-metrics-client-ca\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.472835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.472785 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.474483 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.474457 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.479025 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.474869 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.479025 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.474973 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f63d001-e301-4d65-b409-cda9932c4ea3-node-exporter-tls\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.485985 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.485957 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hf4j\" (UniqueName: \"kubernetes.io/projected/0f63d001-e301-4d65-b409-cda9932c4ea3-kube-api-access-6hf4j\") pod \"node-exporter-564pg\" (UID: \"0f63d001-e301-4d65-b409-cda9932c4ea3\") " pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.486095 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.486022 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9jxg\" (UniqueName: \"kubernetes.io/projected/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-kube-api-access-f9jxg\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.508485 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.508443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.609575 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.609536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-564pg" Jul 06 15:35:14.611339 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.611315 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.622607 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:35:14.622582 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f63d001_e301_4d65_b409_cda9932c4ea3.slice/crio-3d14f945b9ebabe3f6958bbc78e0fbb1599bdc3a5d37262c7a73c2d4755f260d WatchSource:0}: Error finding container 3d14f945b9ebabe3f6958bbc78e0fbb1599bdc3a5d37262c7a73c2d4755f260d: Status 404 returned error can't find the container with id 3d14f945b9ebabe3f6958bbc78e0fbb1599bdc3a5d37262c7a73c2d4755f260d Jul 06 15:35:14.714865 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.714819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:14.946150 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.946112 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-564pg" event={"ID":"0f63d001-e301-4d65-b409-cda9932c4ea3","Type":"ContainerStarted","Data":"3d14f945b9ebabe3f6958bbc78e0fbb1599bdc3a5d37262c7a73c2d4755f260d"} Jul 06 15:35:14.976597 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.976565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:14.979153 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:14.979129 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-66zd9\" (UID: \"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:15.021182 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.021114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:15.161694 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.161660 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" Jul 06 15:35:15.209658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.209575 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:15.316551 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.316516 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9"] Jul 06 15:35:15.318722 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.318450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:15.318722 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.318701 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:35:15.321654 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:35:15.321621 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0626e8_8b36_40b7_8d1a_3a3626d4bfdb.slice/crio-7b87df35210759eba8196c932a37654c0803a457b0306ecb215a690919881246 WatchSource:0}: Error finding container 7b87df35210759eba8196c932a37654c0803a457b0306ecb215a690919881246: Status 404 returned error can't find the container with id 7b87df35210759eba8196c932a37654c0803a457b0306ecb215a690919881246 Jul 06 15:35:15.322846 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.322796 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.325639 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.325614 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 15:35:15.325639 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.325632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 15:35:15.325793 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.325672 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 15:35:15.326063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.326001 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 15:35:15.326063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.326027 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 15:35:15.326063 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.326047 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 15:35:15.326412 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.326129 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hk9jz\"" Jul 06 15:35:15.326412 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.326210 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 15:35:15.326494 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.326416 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 15:35:15.326836 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.326816 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 15:35:15.345680 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.345636 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:35:15.380276 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380439 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380439 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380439 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380439 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-config-out\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380469 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-config-volume\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-web-config\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380952 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8g4p\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-kube-api-access-q8g4p\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.380952 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.380739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.431263 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.431191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:15.481660 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.481660 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.481660 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-config-out\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.481927 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-config-volume\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.481927 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.481927 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.482079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-web-config\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.482079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481965 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.482079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.481987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.482079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.482024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8g4p\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-kube-api-access-q8g4p\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.482079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.482052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.482079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.482073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.482422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.482092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.484556 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.484527 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.485210 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.485137 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.485362 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.485225 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.487852 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.487825 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.488104 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.488083 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.489488 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.489456 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.489599 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.489581 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.490199 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.490108 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.490199 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.490125 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-config-volume\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.490199 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.490135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.492436 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.492409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-config-out\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.492830 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.492808 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-web-config\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.493505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.493470 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8g4p\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-kube-api-access-q8g4p\") pod \"alertmanager-main-0\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.634368 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.634327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:35:15.706310 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.706279 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:35:15.907077 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.906997 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:35:15.910232 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:35:15.910198 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778568c1_b447_4cab_aba7_e66134b0bb1d.slice/crio-eaa64991036ee59120430b9fec649b7986819ad5a739bf055b47ec9bb50da426 WatchSource:0}: Error finding container eaa64991036ee59120430b9fec649b7986819ad5a739bf055b47ec9bb50da426: Status 404 returned error can't find the container with id eaa64991036ee59120430b9fec649b7986819ad5a739bf055b47ec9bb50da426 Jul 06 15:35:15.951448 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.951415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerStarted","Data":"eaa64991036ee59120430b9fec649b7986819ad5a739bf055b47ec9bb50da426"} Jul 06 15:35:15.953238 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.953136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" event={"ID":"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb","Type":"ContainerStarted","Data":"75db2d224d1d1128a355fe4aa1a42ce06e8ff0298f35c18157aae7dd17839470"} Jul 06 15:35:15.953238 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.953226 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" event={"ID":"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb","Type":"ContainerStarted","Data":"c081b29cbe4766571e68307a75f50d198869ec2512915c31ce596ca1fbcd8d7e"} Jul 06 15:35:15.953393 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.953256 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" event={"ID":"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb","Type":"ContainerStarted","Data":"7b87df35210759eba8196c932a37654c0803a457b0306ecb215a690919881246"} Jul 06 15:35:15.954623 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.954600 2575 generic.go:356] "Generic (PLEG): container finished" podID="0f63d001-e301-4d65-b409-cda9932c4ea3" containerID="b5360d01f8e46cc38983b11d171ef8f848e52dda1d76bdf203c1f443d8881ef0" exitCode=0 Jul 06 15:35:15.954716 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:15.954672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-564pg" event={"ID":"0f63d001-e301-4d65-b409-cda9932c4ea3","Type":"ContainerDied","Data":"b5360d01f8e46cc38983b11d171ef8f848e52dda1d76bdf203c1f443d8881ef0"} Jul 06 15:35:16.961525 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:16.961414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" event={"ID":"7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb","Type":"ContainerStarted","Data":"95c27845febcbac6a1a6585c6075f6595aeaa40509ec2d198a3a6ecae7a43c71"} Jul 06 15:35:16.963843 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:16.963818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-564pg" event={"ID":"0f63d001-e301-4d65-b409-cda9932c4ea3","Type":"ContainerStarted","Data":"c4d4c6db44c846612ff96def2688230fb1d85a9c6ea430938d4588dc99ca17a4"} Jul 06 15:35:16.963967 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:16.963870 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-564pg" event={"ID":"0f63d001-e301-4d65-b409-cda9932c4ea3","Type":"ContainerStarted","Data":"c3a6aa1632e1d82b69505e1b63457dad3663621aa639a549e0ce70d18c37e6ab"} Jul 06 15:35:16.988953 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:16.988896 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-66zd9" podStartSLOduration=1.731982178 podStartE2EDuration="2.988880535s" totalImagesPullingTime="1.256898357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:16.987873463 +0000 UTC m=+85.922623526" watchObservedRunningTime="2026-07-06 15:35:16.988880535 +0000 UTC m=+85.923630582" Jul 06 15:35:17.056028 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:17.055978 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-564pg" podStartSLOduration=1.810060425 podStartE2EDuration="3.055960231s" totalImagesPullingTime="1.17038849s" totalInitContainerRuntime="75.511316ms" isStatefulPod=true podCreationTimestamp="2026-07-06 15:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:17.054427596 +0000 UTC m=+85.989177656" watchObservedRunningTime="2026-07-06 15:35:17.055960231 +0000 UTC m=+85.990710313" Jul 06 15:35:17.340797 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:17.340758 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:17.540215 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:17.540148 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:17.712155 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:17.712063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:17.958129 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:17.958092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:17.968353 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:17.968271 2575 generic.go:356] "Generic (PLEG): container finished" podID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerID="db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945" exitCode=0 Jul 06 15:35:17.968740 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:17.968357 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945"} Jul 06 15:35:18.170439 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:18.170401 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:18.708760 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:18.708725 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:18.825817 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:18.825775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.033752 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.033722 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.173564 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.173532 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-kj88m"] Jul 06 15:35:19.175792 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.175769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-kj88m" Jul 06 15:35:19.178917 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.178889 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 15:35:19.179042 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.178934 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-8lh8b\"" Jul 06 15:35:19.179042 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.179024 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 15:35:19.203726 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.203695 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-kj88m"] Jul 06 15:35:19.217421 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.217384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.303662 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.303577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.314157 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.314124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5s2t\" (UniqueName: \"kubernetes.io/projected/304631ed-a1bd-4a6e-88bd-d92b8fdbeb48-kube-api-access-d5s2t\") pod \"downloads-685bfb5f64-kj88m\" (UID: \"304631ed-a1bd-4a6e-88bd-d92b8fdbeb48\") " pod="openshift-console/downloads-685bfb5f64-kj88m" Jul 06 15:35:19.415406 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.415370 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5s2t\" (UniqueName: \"kubernetes.io/projected/304631ed-a1bd-4a6e-88bd-d92b8fdbeb48-kube-api-access-d5s2t\") pod \"downloads-685bfb5f64-kj88m\" (UID: \"304631ed-a1bd-4a6e-88bd-d92b8fdbeb48\") " pod="openshift-console/downloads-685bfb5f64-kj88m" Jul 06 15:35:19.431894 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.431865 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5s2t\" (UniqueName: \"kubernetes.io/projected/304631ed-a1bd-4a6e-88bd-d92b8fdbeb48-kube-api-access-d5s2t\") pod \"downloads-685bfb5f64-kj88m\" (UID: \"304631ed-a1bd-4a6e-88bd-d92b8fdbeb48\") " pod="openshift-console/downloads-685bfb5f64-kj88m" Jul 06 15:35:19.438575 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.438533 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.486530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.486494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-kj88m" Jul 06 15:35:19.538455 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.538419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.639254 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.639149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.722446 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.722374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:19.816522 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.816497 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-kj88m"] Jul 06 15:35:19.819642 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:35:19.819616 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304631ed_a1bd_4a6e_88bd_d92b8fdbeb48.slice/crio-ba12e0b8df2c6e8d4471cb3b5453242d96f71243a9a86943537b22ed2cb62c59 WatchSource:0}: Error finding container ba12e0b8df2c6e8d4471cb3b5453242d96f71243a9a86943537b22ed2cb62c59: Status 404 returned error can't find the container with id ba12e0b8df2c6e8d4471cb3b5453242d96f71243a9a86943537b22ed2cb62c59 Jul 06 15:35:19.977411 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.977377 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-kj88m" event={"ID":"304631ed-a1bd-4a6e-88bd-d92b8fdbeb48","Type":"ContainerStarted","Data":"ba12e0b8df2c6e8d4471cb3b5453242d96f71243a9a86943537b22ed2cb62c59"} Jul 06 15:35:19.980374 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.980154 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerStarted","Data":"4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897"} Jul 06 15:35:19.980374 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.980247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerStarted","Data":"d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5"} Jul 06 15:35:19.980374 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.980289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerStarted","Data":"547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83"} Jul 06 15:35:19.980374 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:19.980328 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerStarted","Data":"35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe"} Jul 06 15:35:20.017411 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.017360 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:20.162273 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.162157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:20.320592 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.320552 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:20.720866 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.720812 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" podUID="ff3ffc05-b70d-462c-82b4-dfdfbf102bff" containerName="registry" containerID="cri-o://18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0" gracePeriod=30 Jul 06 15:35:20.811263 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.811230 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:35:20.815338 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.815317 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.818359 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.818337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 15:35:20.818788 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.818774 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 15:35:20.819059 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.819043 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5jdanfgt1kkmp\"" Jul 06 15:35:20.819309 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.819292 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvqx7\"" Jul 06 15:35:20.819885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.819870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 15:35:20.820434 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.820418 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 15:35:20.821314 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821291 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:20.821421 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821339 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 15:35:20.821421 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821347 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 15:35:20.821421 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821384 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 15:35:20.821603 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821423 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 15:35:20.821603 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821541 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 15:35:20.821603 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821558 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 15:35:20.821774 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821761 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 15:35:20.821882 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 15:35:20.821934 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.821918 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 15:35:20.822546 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.822530 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 15:35:20.823518 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.823498 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 15:35:20.835922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.835902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:35:20.928713 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.928678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.928865 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.928745 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.928865 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.928782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.928865 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.928817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4skd\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-kube-api-access-j4skd\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929062 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.928881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929062 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.928926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929062 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.928974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929062 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929065 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929221 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929323 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929387 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.929909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.929484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:20.981322 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.981252 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:35:20.986886 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.986857 2575 generic.go:356] "Generic (PLEG): container finished" podID="ff3ffc05-b70d-462c-82b4-dfdfbf102bff" containerID="18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0" exitCode=0 Jul 06 15:35:20.987037 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.986897 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" Jul 06 15:35:20.987037 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.986942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" event={"ID":"ff3ffc05-b70d-462c-82b4-dfdfbf102bff","Type":"ContainerDied","Data":"18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0"} Jul 06 15:35:20.987037 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.987004 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dd9b99d4b-nlgf4" event={"ID":"ff3ffc05-b70d-462c-82b4-dfdfbf102bff","Type":"ContainerDied","Data":"218f63ebef01d1a0679fa48a92df6aba5437bf7a5716108a77bc4574a2e94707"} Jul 06 15:35:20.987153 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.987036 2575 scope.go:117] "RemoveContainer" containerID="18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0" Jul 06 15:35:20.998241 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.998208 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerStarted","Data":"3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65"} Jul 06 15:35:20.998430 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:20.998310 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerStarted","Data":"10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c"} Jul 06 15:35:21.006883 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.006848 2575 scope.go:117] "RemoveContainer" containerID="18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0" Jul 06 15:35:21.007416 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:35:21.007320 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0\": container with ID starting with 18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0 not found: ID does not exist" containerID="18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0" Jul 06 15:35:21.007416 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.007359 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0"} err="failed to get container status \"18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0\": rpc error: code = NotFound desc = could not find container \"18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0\": container with ID starting with 18f44c17d5879bdb9dd06cdecebc33fb5de1bc6ef802789854cf53666a70c0f0 not found: ID does not exist" Jul 06 15:35:21.022425 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.022390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:21.029764 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.029735 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-image-registry-private-configuration\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.029922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.029792 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-installation-pull-secrets\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.029922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.029827 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-bound-sa-token\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.029922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.029866 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-certificates\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.029922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.029896 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt99f\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-kube-api-access-mt99f\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.030117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.029948 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-trusted-ca\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.030117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.029976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-ca-trust-extracted\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.030117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030022 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") pod \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\" (UID: \"ff3ffc05-b70d-462c-82b4-dfdfbf102bff\") " Jul 06 15:35:21.030287 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030287 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030253 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030379 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030379 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4skd\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-kube-api-access-j4skd\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030379 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030394 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030440 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:35:21.030529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030446 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030543 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030639 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.030765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.031462 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.031462 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.030902 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-certificates\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.031462 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.031041 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.033211 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.033185 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.034347 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.034322 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.035839 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.035143 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.035839 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.035522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.036527 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.036501 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:35:21.036726 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.036707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.037318 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.037185 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.037492 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.037412 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.037827 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.037593 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-kube-api-access-mt99f" (OuterVolumeSpecName: "kube-api-access-mt99f") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "kube-api-access-mt99f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:35:21.038028 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.037904 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.038366 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.038342 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.038733 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.038675 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:35:21.038826 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.038787 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.039015 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.038992 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.040141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.040102 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.040416 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.040396 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.040678 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.040631 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.041583 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.041381 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:35:21.042221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.041839 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.042221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.041935 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.043021 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.042664 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:35:21.043021 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.042906 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:35:21.044034 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.043321 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.044434 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.044376 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.045747 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.045721 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4skd\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-kube-api-access-j4skd\") pod \"prometheus-k8s-0\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.047370 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.047347 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ff3ffc05-b70d-462c-82b4-dfdfbf102bff" (UID: "ff3ffc05-b70d-462c-82b4-dfdfbf102bff"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:35:21.066315 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.066253 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.3416524450000002 podStartE2EDuration="6.066230827s" totalImagesPullingTime="3.708529379s" totalInitContainerRuntime="16.049003ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:21.062744034 +0000 UTC m=+89.997494081" watchObservedRunningTime="2026-07-06 15:35:21.066230827 +0000 UTC m=+90.000980876" Jul 06 15:35:21.130731 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.130684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:21.132568 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.132531 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-registry-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.132707 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.132661 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-image-registry-private-configuration\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.132707 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.132678 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-installation-pull-secrets\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.132707 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.132694 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-bound-sa-token\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.132865 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.132709 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt99f\" (UniqueName: \"kubernetes.io/projected/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-kube-api-access-mt99f\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.132865 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.132724 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-trusted-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.132865 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.132737 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff3ffc05-b70d-462c-82b4-dfdfbf102bff-ca-trust-extracted\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:35:21.149256 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.149223 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:21.320184 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.320128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:21.325837 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.325808 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-dd9b99d4b-nlgf4"] Jul 06 15:35:21.328994 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.328967 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-dd9b99d4b-nlgf4"] Jul 06 15:35:21.333970 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.333885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:35:21.338944 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:35:21.338915 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb42d9f_9c95_4f62_b91e_1a3d7c216a2b.slice/crio-263ccf9f5d3b8198899f227d4366a6575a42eb8a43016c37a295543081032e98 WatchSource:0}: Error finding container 263ccf9f5d3b8198899f227d4366a6575a42eb8a43016c37a295543081032e98: Status 404 returned error can't find the container with id 263ccf9f5d3b8198899f227d4366a6575a42eb8a43016c37a295543081032e98 Jul 06 15:35:21.584868 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.584780 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:21.604310 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.604269 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff3ffc05-b70d-462c-82b4-dfdfbf102bff" path="/var/lib/kubelet/pods/ff3ffc05-b70d-462c-82b4-dfdfbf102bff/volumes" Jul 06 15:35:21.845444 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.845351 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:21.913296 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:21.913264 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:22.004316 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:22.004277 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerID="186b764daff601b2850c0720c8f8209292a33bf217134901d767ac79f35f825f" exitCode=0 Jul 06 15:35:22.004496 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:22.004378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"186b764daff601b2850c0720c8f8209292a33bf217134901d767ac79f35f825f"} Jul 06 15:35:22.004496 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:22.004465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerStarted","Data":"263ccf9f5d3b8198899f227d4366a6575a42eb8a43016c37a295543081032e98"} Jul 06 15:35:26.026642 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:26.026558 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerStarted","Data":"0706637efc8e506d6af35add1cafd0c371342cff51d7b34181379652756d8b02"} Jul 06 15:35:26.027015 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:26.026654 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerStarted","Data":"5dc5be2760f1043359cf520697bed897a12fd0842d70a06dfc6fc1a13b0b69c8"} Jul 06 15:35:27.726682 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:27.726635 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:27.862759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:27.862680 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:27.973079 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:27.973024 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:29.045040 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:29.045001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerStarted","Data":"2b1a3045380e23a8c8d30544dc9bb9835da5563491b0ae0386456b2a1de9820b"} Jul 06 15:35:29.045477 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:29.045080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerStarted","Data":"21fb8d52caa8149689d463d461257442b568a5b27d77bb720984b76c2ae366d3"} Jul 06 15:35:29.045477 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:29.045121 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerStarted","Data":"96b2d501b51879a84f30f32aac52225289f16e4cf29ccb41c2b89db4c452bd05"} Jul 06 15:35:29.045477 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:29.045183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerStarted","Data":"24de674bfeaa708e68fdfa9e9c70593b37bbcef6de016434abc2b75edb1b20ff"} Jul 06 15:35:29.085968 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:29.085903 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.082616679 podStartE2EDuration="9.085886502s" totalImagesPullingTime="5.976847723s" totalInitContainerRuntime="26.4221ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:29.083540453 +0000 UTC m=+98.018290500" watchObservedRunningTime="2026-07-06 15:35:29.085886502 +0000 UTC m=+98.020636556" Jul 06 15:35:30.498451 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:30.498414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:30.568873 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:30.568838 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:31.131239 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:31.131208 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:35:33.255457 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:33.255368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:33.363567 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:33.363534 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:34.159913 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:34.159869 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:34.262175 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:34.262116 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:34.904323 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:34.904292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2446q" Jul 06 15:35:36.010348 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:36.010312 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:36.071658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:36.071610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:37.075157 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:37.075115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:40.092696 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:40.092653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-kj88m" event={"ID":"304631ed-a1bd-4a6e-88bd-d92b8fdbeb48","Type":"ContainerStarted","Data":"f281ad2c82817012975e34a17a716bb5a64db174d6f7796b4ffaf0b6c75df654"} Jul 06 15:35:40.093212 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:40.092881 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-kj88m" Jul 06 15:35:40.112308 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:40.112276 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-kj88m" Jul 06 15:35:40.116771 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:40.116722 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-kj88m" podStartSLOduration=1.420131983 podStartE2EDuration="21.116705263s" totalImagesPullingTime="19.69657328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:35:40.115048638 +0000 UTC m=+109.049798694" watchObservedRunningTime="2026-07-06 15:35:40.116705263 +0000 UTC m=+109.051455371" Jul 06 15:35:43.311585 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:43.311550 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778568c1-b447-4cab-aba7-e66134b0bb1d/init-config-reloader/0.log" Jul 06 15:35:43.319149 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:43.319118 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778568c1-b447-4cab-aba7-e66134b0bb1d/alertmanager/0.log" Jul 06 15:35:43.409015 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:43.408981 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778568c1-b447-4cab-aba7-e66134b0bb1d/config-reloader/0.log" Jul 06 15:35:43.628509 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:43.628432 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778568c1-b447-4cab-aba7-e66134b0bb1d/kube-rbac-proxy-web/0.log" Jul 06 15:35:43.809879 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:43.809847 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778568c1-b447-4cab-aba7-e66134b0bb1d/kube-rbac-proxy/0.log" Jul 06 15:35:44.007933 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:44.007850 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778568c1-b447-4cab-aba7-e66134b0bb1d/kube-rbac-proxy-metric/0.log" Jul 06 15:35:44.208033 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:44.207992 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_778568c1-b447-4cab-aba7-e66134b0bb1d/prom-label-proxy/0.log" Jul 06 15:35:46.012090 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:46.012055 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-564pg_0f63d001-e301-4d65-b409-cda9932c4ea3/init-textfile/0.log" Jul 06 15:35:46.211151 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:46.211112 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-564pg_0f63d001-e301-4d65-b409-cda9932c4ea3/node-exporter/0.log" Jul 06 15:35:46.278240 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:46.278140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:46.351047 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:46.351006 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:46.408667 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:46.408633 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-564pg_0f63d001-e301-4d65-b409-cda9932c4ea3/kube-rbac-proxy/0.log" Jul 06 15:35:47.179741 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:47.179710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:47.437334 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:47.437257 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:47.809435 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:47.809397 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-66zd9_7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb/kube-rbac-proxy-main/0.log" Jul 06 15:35:48.010503 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:48.010473 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-66zd9_7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb/kube-rbac-proxy-self/0.log" Jul 06 15:35:48.210449 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:48.210358 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-66zd9_7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb/openshift-state-metrics/0.log" Jul 06 15:35:48.409606 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:48.409576 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/init-config-reloader/0.log" Jul 06 15:35:48.611826 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:48.611793 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/prometheus/0.log" Jul 06 15:35:48.808412 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:48.808375 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/config-reloader/0.log" Jul 06 15:35:49.008703 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:49.008623 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/thanos-sidecar/0.log" Jul 06 15:35:49.209407 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:49.209378 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/kube-rbac-proxy-web/0.log" Jul 06 15:35:49.409822 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:49.409793 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/kube-rbac-proxy/0.log" Jul 06 15:35:49.608857 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:49.608824 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/kube-rbac-proxy-thanos/0.log" Jul 06 15:35:51.022774 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:51.022740 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:51.122231 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:51.122195 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:35:52.609695 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:35:52.609662 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-kj88m_304631ed-a1bd-4a6e-88bd-d92b8fdbeb48/download-server/0.log" Jul 06 15:36:21.131591 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:21.131545 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:21.187221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:21.187191 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:21.273576 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:21.273552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:23.177246 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:23.177212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:33.798543 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:33.798499 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:34.049848 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.049748 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:34.860410 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.860376 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:36:34.860900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.860662 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="alertmanager" containerID="cri-o://35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe" gracePeriod=120 Jul 06 15:36:34.860900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.860726 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-metric" containerID="cri-o://10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c" gracePeriod=120 Jul 06 15:36:34.860900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.860756 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="config-reloader" containerID="cri-o://547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83" gracePeriod=120 Jul 06 15:36:34.860900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.860795 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy" containerID="cri-o://4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897" gracePeriod=120 Jul 06 15:36:34.860900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.860726 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-web" containerID="cri-o://d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5" gracePeriod=120 Jul 06 15:36:34.860900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:34.860812 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="prom-label-proxy" containerID="cri-o://3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65" gracePeriod=120 Jul 06 15:36:35.309954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.309921 2575 generic.go:356] "Generic (PLEG): container finished" podID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerID="3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65" exitCode=0 Jul 06 15:36:35.309954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.309947 2575 generic.go:356] "Generic (PLEG): container finished" podID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerID="10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c" exitCode=0 Jul 06 15:36:35.309954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.309953 2575 generic.go:356] "Generic (PLEG): container finished" podID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerID="4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897" exitCode=0 Jul 06 15:36:35.309954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.309959 2575 generic.go:356] "Generic (PLEG): container finished" podID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerID="547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83" exitCode=0 Jul 06 15:36:35.309954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.309964 2575 generic.go:356] "Generic (PLEG): container finished" podID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerID="35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe" exitCode=0 Jul 06 15:36:35.310268 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.309988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65"} Jul 06 15:36:35.310268 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.310054 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c"} Jul 06 15:36:35.310268 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.310083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897"} Jul 06 15:36:35.310268 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.310118 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83"} Jul 06 15:36:35.310268 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:35.310142 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe"} Jul 06 15:36:36.120104 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.120077 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.293690 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293661 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-web\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.293877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293706 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-main-db\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.293877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293747 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-metrics-client-ca\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.293877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293776 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-config-volume\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.293877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293795 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.293877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293817 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-config-out\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.293877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293832 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-cluster-tls-config\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.293877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8g4p\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-kube-api-access-q8g4p\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.294266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293920 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.294266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293969 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-main-tls\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.294266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.293998 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-web-config\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.294266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.294023 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-tls-assets\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.294266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.294045 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-trusted-ca-bundle\") pod \"778568c1-b447-4cab-aba7-e66134b0bb1d\" (UID: \"778568c1-b447-4cab-aba7-e66134b0bb1d\") " Jul 06 15:36:36.294266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.294149 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:36.294266 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.294231 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:36:36.294603 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.294365 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-main-db\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.294603 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.294384 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-metrics-client-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.294995 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.294692 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:36.296960 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.296932 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:36.297103 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.296979 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-config-volume" (OuterVolumeSpecName: "config-volume") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:36.297370 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.297247 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-kube-api-access-q8g4p" (OuterVolumeSpecName: "kube-api-access-q8g4p") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "kube-api-access-q8g4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:36:36.297518 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.297497 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:36.297726 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.297708 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:36:36.298106 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.298076 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:36.298201 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.298143 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-config-out" (OuterVolumeSpecName: "config-out") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:36:36.298754 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.298730 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:36.302664 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.302633 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:36.309277 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.309240 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-web-config" (OuterVolumeSpecName: "web-config") pod "778568c1-b447-4cab-aba7-e66134b0bb1d" (UID: "778568c1-b447-4cab-aba7-e66134b0bb1d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:36.316914 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.316887 2575 generic.go:356] "Generic (PLEG): container finished" podID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerID="d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5" exitCode=0 Jul 06 15:36:36.317056 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.316976 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.317056 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.316973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5"} Jul 06 15:36:36.317193 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.317075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"778568c1-b447-4cab-aba7-e66134b0bb1d","Type":"ContainerDied","Data":"eaa64991036ee59120430b9fec649b7986819ad5a739bf055b47ec9bb50da426"} Jul 06 15:36:36.317193 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.317131 2575 scope.go:117] "RemoveContainer" containerID="3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65" Jul 06 15:36:36.325268 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.325245 2575 scope.go:117] "RemoveContainer" containerID="10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c" Jul 06 15:36:36.332325 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.332305 2575 scope.go:117] "RemoveContainer" containerID="4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897" Jul 06 15:36:36.339032 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.339013 2575 scope.go:117] "RemoveContainer" containerID="d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5" Jul 06 15:36:36.343193 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.343145 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:36:36.348484 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.347422 2575 scope.go:117] "RemoveContainer" containerID="547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83" Jul 06 15:36:36.354128 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.354098 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:36:36.359962 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.359938 2575 scope.go:117] "RemoveContainer" containerID="35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe" Jul 06 15:36:36.367442 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.367417 2575 scope.go:117] "RemoveContainer" containerID="db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945" Jul 06 15:36:36.374930 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.374912 2575 scope.go:117] "RemoveContainer" containerID="3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65" Jul 06 15:36:36.375256 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.375233 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65\": container with ID starting with 3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65 not found: ID does not exist" containerID="3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65" Jul 06 15:36:36.375352 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.375269 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65"} err="failed to get container status \"3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65\": rpc error: code = NotFound desc = could not find container \"3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65\": container with ID starting with 3684ba561792e41039e5a35ad14554d2ab4fda524149d0d26d39b9f902438d65 not found: ID does not exist" Jul 06 15:36:36.375352 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.375293 2575 scope.go:117] "RemoveContainer" containerID="10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c" Jul 06 15:36:36.375537 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.375519 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c\": container with ID starting with 10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c not found: ID does not exist" containerID="10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c" Jul 06 15:36:36.375573 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.375544 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c"} err="failed to get container status \"10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c\": rpc error: code = NotFound desc = could not find container \"10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c\": container with ID starting with 10b003c51e18fd50e8cc96aa126deb50d1e348a56406f06dfd71a8631b85c50c not found: ID does not exist" Jul 06 15:36:36.375573 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.375566 2575 scope.go:117] "RemoveContainer" containerID="4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897" Jul 06 15:36:36.375792 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.375775 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897\": container with ID starting with 4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897 not found: ID does not exist" containerID="4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897" Jul 06 15:36:36.375835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.375798 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897"} err="failed to get container status \"4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897\": rpc error: code = NotFound desc = could not find container \"4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897\": container with ID starting with 4e8be6d74e0af021788ceee0f134155a5b8ae7d2e78c2aba9f70a514e0445897 not found: ID does not exist" Jul 06 15:36:36.375835 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.375815 2575 scope.go:117] "RemoveContainer" containerID="d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5" Jul 06 15:36:36.376036 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.376022 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5\": container with ID starting with d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5 not found: ID does not exist" containerID="d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5" Jul 06 15:36:36.376081 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.376039 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5"} err="failed to get container status \"d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5\": rpc error: code = NotFound desc = could not find container \"d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5\": container with ID starting with d2afa02321db682414d909376cbceff7a19361a000bd9e60673b04d9e8da91f5 not found: ID does not exist" Jul 06 15:36:36.376081 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.376049 2575 scope.go:117] "RemoveContainer" containerID="547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83" Jul 06 15:36:36.376261 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.376244 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83\": container with ID starting with 547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83 not found: ID does not exist" containerID="547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83" Jul 06 15:36:36.376305 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.376264 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83"} err="failed to get container status \"547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83\": rpc error: code = NotFound desc = could not find container \"547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83\": container with ID starting with 547c962b7ac39f17b09a70ea6a29b81690abf93869b3e2592d51cb7b82b1cf83 not found: ID does not exist" Jul 06 15:36:36.376305 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.376275 2575 scope.go:117] "RemoveContainer" containerID="35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe" Jul 06 15:36:36.376592 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.376539 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe\": container with ID starting with 35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe not found: ID does not exist" containerID="35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe" Jul 06 15:36:36.376708 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.376595 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe"} err="failed to get container status \"35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe\": rpc error: code = NotFound desc = could not find container \"35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe\": container with ID starting with 35cfdd5a3fd2bbec40745db1d93756ab424a22785705aa0fb23d122b06de55fe not found: ID does not exist" Jul 06 15:36:36.376708 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.376618 2575 scope.go:117] "RemoveContainer" containerID="db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945" Jul 06 15:36:36.376883 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.376857 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945\": container with ID starting with db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945 not found: ID does not exist" containerID="db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945" Jul 06 15:36:36.376998 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.376888 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945"} err="failed to get container status \"db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945\": rpc error: code = NotFound desc = could not find container \"db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945\": container with ID starting with db24e4b6c72ff9ae331232729a52a4647e961682f397f405e373e33bcf1dd945 not found: ID does not exist" Jul 06 15:36:36.378528 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378508 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:36:36.378793 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378781 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="init-config-reloader" Jul 06 15:36:36.378833 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378794 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="init-config-reloader" Jul 06 15:36:36.378833 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378807 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-web" Jul 06 15:36:36.378833 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378812 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-web" Jul 06 15:36:36.378833 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378818 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy" Jul 06 15:36:36.378833 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378823 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy" Jul 06 15:36:36.378833 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378830 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="config-reloader" Jul 06 15:36:36.378833 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378835 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="config-reloader" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378842 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="alertmanager" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378847 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="alertmanager" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378853 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="prom-label-proxy" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378858 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="prom-label-proxy" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378869 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-metric" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378874 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-metric" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378881 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff3ffc05-b70d-462c-82b4-dfdfbf102bff" containerName="registry" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378886 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff3ffc05-b70d-462c-82b4-dfdfbf102bff" containerName="registry" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378942 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-metric" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378950 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378956 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff3ffc05-b70d-462c-82b4-dfdfbf102bff" containerName="registry" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378961 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="alertmanager" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378970 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="prom-label-proxy" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378976 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="config-reloader" Jul 06 15:36:36.379029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.378980 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" containerName="kube-rbac-proxy-web" Jul 06 15:36:36.382334 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.382318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.384992 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.384973 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 06 15:36:36.385114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.385087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hk9jz\"" Jul 06 15:36:36.386052 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.386036 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 06 15:36:36.386052 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.386049 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 06 15:36:36.386204 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.386064 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 06 15:36:36.386513 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.386499 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 06 15:36:36.386597 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.386527 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 06 15:36:36.387001 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.386983 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 06 15:36:36.388313 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.388295 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 06 15:36:36.393468 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.393448 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 06 15:36:36.395510 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395494 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-main-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395530 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-web-config\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395541 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-tls-assets\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395549 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/778568c1-b447-4cab-aba7-e66134b0bb1d-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395559 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395567 2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-config-volume\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395575 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395583 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778568c1-b447-4cab-aba7-e66134b0bb1d-config-out\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395592 2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-cluster-tls-config\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395601 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8g4p\" (UniqueName: \"kubernetes.io/projected/778568c1-b447-4cab-aba7-e66134b0bb1d-kube-api-access-q8g4p\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.395909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.395610 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/778568c1-b447-4cab-aba7-e66134b0bb1d-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:36.402288 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.402260 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:36:36.496970 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.496937 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497180 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.496983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497180 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497007 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497180 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vt8g\" (UniqueName: \"kubernetes.io/projected/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-kube-api-access-5vt8g\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497180 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497180 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497383 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497189 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-config-out\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497383 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497208 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497383 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497383 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497383 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497278 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-web-config\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497383 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497324 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.497383 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.497364 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598059 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.597957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598059 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598059 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-web-config\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598306 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598306 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598306 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.598450 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.598387 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vt8g\" (UniqueName: \"kubernetes.io/projected/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-kube-api-access-5vt8g\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.599786 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.599758 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.599982 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.599964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.600102 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.600087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.600244 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.600226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-config-out\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.600359 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.600340 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.601489 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.601458 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.601597 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.601553 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.601670 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.601601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.601670 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.601628 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.601784 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.601678 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-web-config\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.601784 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.601711 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-config-volume\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.601784 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.601758 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.602064 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.602044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.602114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.602086 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.602746 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.602725 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.603620 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.603603 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-config-out\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.611046 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.611028 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vt8g\" (UniqueName: \"kubernetes.io/projected/2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69-kube-api-access-5vt8g\") pod \"alertmanager-main-0\" (UID: \"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69\") " pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.692759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.692718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 06 15:36:36.860655 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:36.860574 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 06 15:36:36.875016 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:36:36.874984 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4aa4fb_d496_4fc8_8ee0_64ed3e5b0e69.slice/crio-b38cba947b74a11039b2766ff6c81d8e232610dbf3d368c67e76344f41b40f2e WatchSource:0}: Error finding container b38cba947b74a11039b2766ff6c81d8e232610dbf3d368c67e76344f41b40f2e: Status 404 returned error can't find the container with id b38cba947b74a11039b2766ff6c81d8e232610dbf3d368c67e76344f41b40f2e Jul 06 15:36:36.995452 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:36:36.995417 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4aa4fb_d496_4fc8_8ee0_64ed3e5b0e69.slice/crio-conmon-a9ef7dbfb09333a1404b40f99dd87ed728926654919fb7781d3941520fe1655e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4aa4fb_d496_4fc8_8ee0_64ed3e5b0e69.slice/crio-a9ef7dbfb09333a1404b40f99dd87ed728926654919fb7781d3941520fe1655e.scope\": RecentStats: unable to find data in memory cache]" Jul 06 15:36:37.321452 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:37.321413 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69" containerID="a9ef7dbfb09333a1404b40f99dd87ed728926654919fb7781d3941520fe1655e" exitCode=0 Jul 06 15:36:37.321922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:37.321506 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerDied","Data":"a9ef7dbfb09333a1404b40f99dd87ed728926654919fb7781d3941520fe1655e"} Jul 06 15:36:37.321922 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:37.321583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerStarted","Data":"b38cba947b74a11039b2766ff6c81d8e232610dbf3d368c67e76344f41b40f2e"} Jul 06 15:36:37.603093 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:37.603067 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778568c1-b447-4cab-aba7-e66134b0bb1d" path="/var/lib/kubelet/pods/778568c1-b447-4cab-aba7-e66134b0bb1d/volumes" Jul 06 15:36:38.334743 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.334711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerStarted","Data":"6a8b15adc2231a987077dac75f9b9f0c1505dac7d4cc1b167a11a86e93a49a59"} Jul 06 15:36:38.335132 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.334798 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerStarted","Data":"f8d4b7d4b3f2262933a80316969463969f3706cbc33d1591ddfcb03992e3b1ee"} Jul 06 15:36:38.335132 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.334824 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerStarted","Data":"30ee57b1b7abbae7f06a1e1b7f00712c21dcbca0a7d700a9248d47ddc73539e1"} Jul 06 15:36:38.335132 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.334848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerStarted","Data":"c823e27cfcce13536d2a87eddd29847655044a6d8fa2b0d438ef1bc5c4f5aa4a"} Jul 06 15:36:38.335132 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.334875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerStarted","Data":"447639a54583b28ce9949a1941529dd99bbd9fbecc9ca77c59f0d9265235accb"} Jul 06 15:36:38.335132 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.334900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69","Type":"ContainerStarted","Data":"4b6dcdf2e04fdc494521daee8abaf252ccb774731287782891e4754b1fff89e8"} Jul 06 15:36:38.374400 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.374339 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.359077593 podStartE2EDuration="2.374322187s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.244594ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:36:38.372146648 +0000 UTC m=+167.306896696" watchObservedRunningTime="2026-07-06 15:36:38.374322187 +0000 UTC m=+167.309072233" Jul 06 15:36:38.926666 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.926632 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-64dfc4c9-5bjwk"] Jul 06 15:36:38.928972 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.928956 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:38.931880 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.931854 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-z4fbb\"" Jul 06 15:36:38.932023 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.931921 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 06 15:36:38.932023 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.931976 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 06 15:36:38.932023 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.931979 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 06 15:36:38.932277 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.932156 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 06 15:36:38.932408 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.932289 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 06 15:36:38.938008 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.937989 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 06 15:36:38.946417 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:38.946391 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64dfc4c9-5bjwk"] Jul 06 15:36:39.025573 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025533 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-secret-telemeter-client\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.025573 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025578 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-telemeter-client-tls\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.025803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.025803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025663 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8g7r\" (UniqueName: \"kubernetes.io/projected/f29da444-c54c-44aa-876b-f02a6c5f05d9-kube-api-access-b8g7r\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.025803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-metrics-client-ca\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.025803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-federate-client-tls\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.025803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.025803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.025799 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-serving-certs-ca-bundle\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.126909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.126866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8g7r\" (UniqueName: \"kubernetes.io/projected/f29da444-c54c-44aa-876b-f02a6c5f05d9-kube-api-access-b8g7r\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127057 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.126926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-metrics-client-ca\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127057 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.126972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-federate-client-tls\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127057 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.127004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127057 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.127030 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-serving-certs-ca-bundle\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127329 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.127082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-secret-telemeter-client\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127329 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.127114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-telemeter-client-tls\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127329 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.127139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127798 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.127768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-metrics-client-ca\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.127938 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.127904 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-serving-certs-ca-bundle\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.128216 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.128193 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f29da444-c54c-44aa-876b-f02a6c5f05d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.129742 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.129724 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-federate-client-tls\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.129892 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.129875 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.130270 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.130249 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-telemeter-client-tls\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.130335 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.130270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f29da444-c54c-44aa-876b-f02a6c5f05d9-secret-telemeter-client\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.135906 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.135882 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8g7r\" (UniqueName: \"kubernetes.io/projected/f29da444-c54c-44aa-876b-f02a6c5f05d9-kube-api-access-b8g7r\") pod \"telemeter-client-64dfc4c9-5bjwk\" (UID: \"f29da444-c54c-44aa-876b-f02a6c5f05d9\") " pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.205882 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.205793 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:36:39.206114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.206072 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="prometheus" containerID="cri-o://5dc5be2760f1043359cf520697bed897a12fd0842d70a06dfc6fc1a13b0b69c8" gracePeriod=600 Jul 06 15:36:39.206221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.206111 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="thanos-sidecar" containerID="cri-o://24de674bfeaa708e68fdfa9e9c70593b37bbcef6de016434abc2b75edb1b20ff" gracePeriod=600 Jul 06 15:36:39.206221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.206142 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2b1a3045380e23a8c8d30544dc9bb9835da5563491b0ae0386456b2a1de9820b" gracePeriod=600 Jul 06 15:36:39.206221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.206147 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="config-reloader" containerID="cri-o://0706637efc8e506d6af35add1cafd0c371342cff51d7b34181379652756d8b02" gracePeriod=600 Jul 06 15:36:39.206221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.206195 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-web" containerID="cri-o://96b2d501b51879a84f30f32aac52225289f16e4cf29ccb41c2b89db4c452bd05" gracePeriod=600 Jul 06 15:36:39.206389 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.206111 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy" containerID="cri-o://21fb8d52caa8149689d463d461257442b568a5b27d77bb720984b76c2ae366d3" gracePeriod=600 Jul 06 15:36:39.239868 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.239845 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" Jul 06 15:36:39.345497 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.345459 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerID="2b1a3045380e23a8c8d30544dc9bb9835da5563491b0ae0386456b2a1de9820b" exitCode=0 Jul 06 15:36:39.345950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.345593 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerID="21fb8d52caa8149689d463d461257442b568a5b27d77bb720984b76c2ae366d3" exitCode=0 Jul 06 15:36:39.345950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.345620 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerID="96b2d501b51879a84f30f32aac52225289f16e4cf29ccb41c2b89db4c452bd05" exitCode=0 Jul 06 15:36:39.345950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.345629 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerID="24de674bfeaa708e68fdfa9e9c70593b37bbcef6de016434abc2b75edb1b20ff" exitCode=0 Jul 06 15:36:39.345950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.345639 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerID="0706637efc8e506d6af35add1cafd0c371342cff51d7b34181379652756d8b02" exitCode=0 Jul 06 15:36:39.345950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.345648 2575 generic.go:356] "Generic (PLEG): container finished" podID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerID="5dc5be2760f1043359cf520697bed897a12fd0842d70a06dfc6fc1a13b0b69c8" exitCode=0 Jul 06 15:36:39.345950 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.345501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"2b1a3045380e23a8c8d30544dc9bb9835da5563491b0ae0386456b2a1de9820b"} Jul 06 15:36:39.346275 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.346021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"21fb8d52caa8149689d463d461257442b568a5b27d77bb720984b76c2ae366d3"} Jul 06 15:36:39.346275 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.346103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"96b2d501b51879a84f30f32aac52225289f16e4cf29ccb41c2b89db4c452bd05"} Jul 06 15:36:39.346275 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.346202 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"24de674bfeaa708e68fdfa9e9c70593b37bbcef6de016434abc2b75edb1b20ff"} Jul 06 15:36:39.346275 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.346243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"0706637efc8e506d6af35add1cafd0c371342cff51d7b34181379652756d8b02"} Jul 06 15:36:39.346467 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.346304 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"5dc5be2760f1043359cf520697bed897a12fd0842d70a06dfc6fc1a13b0b69c8"} Jul 06 15:36:39.387778 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.387750 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-64dfc4c9-5bjwk"] Jul 06 15:36:39.392499 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:36:39.392460 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29da444_c54c_44aa_876b_f02a6c5f05d9.slice/crio-1e8ac5104b4fe4e8fa3f1fd72bdb09e51467d3d6e547c01ebcaf4aa7ce743d72 WatchSource:0}: Error finding container 1e8ac5104b4fe4e8fa3f1fd72bdb09e51467d3d6e547c01ebcaf4aa7ce743d72: Status 404 returned error can't find the container with id 1e8ac5104b4fe4e8fa3f1fd72bdb09e51467d3d6e547c01ebcaf4aa7ce743d72 Jul 06 15:36:39.460697 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.460633 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:39.632188 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632130 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-trusted-ca-bundle\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632188 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632180 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-web-config\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632206 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-1\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632229 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632257 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-db\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-tls-assets\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632300 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-0\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632671 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632644 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:39.632727 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632661 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:39.632770 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632723 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632810 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632769 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-2\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632810 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-kubelet-serving-ca-bundle\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632841 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-tls\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632877 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-metrics-client-ca\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.632909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632902 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-thanos-prometheus-http-client-file\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633069 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.632998 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-kube-rbac-proxy\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633181 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633144 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config-out\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633256 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633199 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-grpc-tls\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633256 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633235 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633270 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4skd\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-kube-api-access-j4skd\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633307 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-serving-certs-ca-bundle\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633358 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633333 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-metrics-client-certs\") pod \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\" (UID: \"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b\") " Jul 06 15:36:39.633507 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633381 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:39.633663 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633640 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.633727 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633669 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.633727 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633686 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.633999 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.633972 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:36:39.634884 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.634643 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:39.634884 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.634737 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:39.635058 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.634987 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:39.635135 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.635113 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 15:36:39.635884 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.635785 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.635884 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.635859 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config" (OuterVolumeSpecName: "config") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.636118 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.636079 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.636679 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.636648 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config-out" (OuterVolumeSpecName: "config-out") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:36:39.637590 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.637560 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-kube-api-access-j4skd" (OuterVolumeSpecName: "kube-api-access-j4skd") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "kube-api-access-j4skd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:36:39.637686 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.637662 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.637756 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.637690 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.637756 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.637705 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.638058 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.638029 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.638463 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.638445 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:36:39.638600 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.638579 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.646024 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.646001 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-web-config" (OuterVolumeSpecName: "web-config") pod "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" (UID: "ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 15:36:39.735084 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.734998 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735084 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735035 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-metrics-client-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735084 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735051 2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735084 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735070 2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-kube-rbac-proxy\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735084 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735084 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config-out\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735096 2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-grpc-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735109 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735121 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4skd\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-kube-api-access-j4skd\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735135 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735149 2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-metrics-client-certs\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735181 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-web-config\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735196 2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-config\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735207 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-db\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735221 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-tls-assets\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735238 2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735252 2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:39.735382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:39.735266 2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:36:40.352810 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.352774 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" event={"ID":"f29da444-c54c-44aa-876b-f02a6c5f05d9","Type":"ContainerStarted","Data":"1e8ac5104b4fe4e8fa3f1fd72bdb09e51467d3d6e547c01ebcaf4aa7ce743d72"} Jul 06 15:36:40.355688 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.355658 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b","Type":"ContainerDied","Data":"263ccf9f5d3b8198899f227d4366a6575a42eb8a43016c37a295543081032e98"} Jul 06 15:36:40.360689 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.356042 2575 scope.go:117] "RemoveContainer" containerID="2b1a3045380e23a8c8d30544dc9bb9835da5563491b0ae0386456b2a1de9820b" Jul 06 15:36:40.360689 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.356142 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.369591 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.369565 2575 scope.go:117] "RemoveContainer" containerID="21fb8d52caa8149689d463d461257442b568a5b27d77bb720984b76c2ae366d3" Jul 06 15:36:40.376647 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.376626 2575 scope.go:117] "RemoveContainer" containerID="96b2d501b51879a84f30f32aac52225289f16e4cf29ccb41c2b89db4c452bd05" Jul 06 15:36:40.384333 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.384312 2575 scope.go:117] "RemoveContainer" containerID="24de674bfeaa708e68fdfa9e9c70593b37bbcef6de016434abc2b75edb1b20ff" Jul 06 15:36:40.384841 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.384817 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:36:40.389567 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.389514 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:36:40.391783 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.391768 2575 scope.go:117] "RemoveContainer" containerID="0706637efc8e506d6af35add1cafd0c371342cff51d7b34181379652756d8b02" Jul 06 15:36:40.398544 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.398522 2575 scope.go:117] "RemoveContainer" containerID="5dc5be2760f1043359cf520697bed897a12fd0842d70a06dfc6fc1a13b0b69c8" Jul 06 15:36:40.405880 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.405866 2575 scope.go:117] "RemoveContainer" containerID="186b764daff601b2850c0720c8f8209292a33bf217134901d767ac79f35f825f" Jul 06 15:36:40.432449 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432420 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:36:40.432716 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432704 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="config-reloader" Jul 06 15:36:40.432759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432718 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="config-reloader" Jul 06 15:36:40.432759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432725 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-thanos" Jul 06 15:36:40.432759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432730 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-thanos" Jul 06 15:36:40.432759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432736 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="prometheus" Jul 06 15:36:40.432759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432742 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="prometheus" Jul 06 15:36:40.432759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432751 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="init-config-reloader" Jul 06 15:36:40.432759 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432756 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="init-config-reloader" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432763 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-web" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432768 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-web" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432775 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="thanos-sidecar" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432779 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="thanos-sidecar" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432788 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432793 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432841 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432848 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-web" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432853 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="config-reloader" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432857 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="kube-rbac-proxy-thanos" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432862 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="thanos-sidecar" Jul 06 15:36:40.432959 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.432868 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" containerName="prometheus" Jul 06 15:36:40.435034 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.435019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.438651 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.438618 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 06 15:36:40.438776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.438733 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5jdanfgt1kkmp\"" Jul 06 15:36:40.438776 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.438755 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvqx7\"" Jul 06 15:36:40.439468 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.439434 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 06 15:36:40.439595 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.439578 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 06 15:36:40.439993 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.439976 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 06 15:36:40.440066 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.439977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 06 15:36:40.440183 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.439980 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 06 15:36:40.440542 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.440524 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 06 15:36:40.440746 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.440729 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 06 15:36:40.441052 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.441035 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 06 15:36:40.441138 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.441089 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 06 15:36:40.441501 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.441481 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 06 15:36:40.442057 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.442035 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 06 15:36:40.443083 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.443064 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 06 15:36:40.443675 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.443657 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 06 15:36:40.448188 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.448147 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 06 15:36:40.459224 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.459200 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:36:40.541412 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541368 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541589 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541589 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541451 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b96d5d40-01bd-4492-bd78-9219c3e1cf13-config-out\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541589 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541589 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541589 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541807 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541807 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrr4c\" (UniqueName: \"kubernetes.io/projected/b96d5d40-01bd-4492-bd78-9219c3e1cf13-kube-api-access-mrr4c\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541807 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-web-config\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541807 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.541807 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-config\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541880 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.541989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542043 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.542016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.542372 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.542049 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b96d5d40-01bd-4492-bd78-9219c3e1cf13-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642619 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642619 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642619 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642587 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b96d5d40-01bd-4492-bd78-9219c3e1cf13-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642619 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642943 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642943 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b96d5d40-01bd-4492-bd78-9219c3e1cf13-config-out\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642943 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642829 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642943 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642862 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642943 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.642943 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643243 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642956 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrr4c\" (UniqueName: \"kubernetes.io/projected/b96d5d40-01bd-4492-bd78-9219c3e1cf13-kube-api-access-mrr4c\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643243 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.642987 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-web-config\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643243 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643243 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-config\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643243 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643243 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643243 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643231 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643572 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643572 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643290 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643572 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643326 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643719 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643610 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643719 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643661 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.643889 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.643866 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.644189 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.644150 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.645532 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.645188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.645973 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.645944 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b96d5d40-01bd-4492-bd78-9219c3e1cf13-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.646081 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.646045 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.646251 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.646227 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.646440 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.646414 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.647793 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.646755 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.647793 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.647479 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.647793 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.647695 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b96d5d40-01bd-4492-bd78-9219c3e1cf13-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.647998 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.647940 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.647998 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.647961 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b96d5d40-01bd-4492-bd78-9219c3e1cf13-config-out\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.648403 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.648316 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.648695 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.648675 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.648933 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.648911 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-config\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.649100 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.649084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.650221 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.650199 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b96d5d40-01bd-4492-bd78-9219c3e1cf13-web-config\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.655607 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.655587 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrr4c\" (UniqueName: \"kubernetes.io/projected/b96d5d40-01bd-4492-bd78-9219c3e1cf13-kube-api-access-mrr4c\") pod \"prometheus-k8s-0\" (UID: \"b96d5d40-01bd-4492-bd78-9219c3e1cf13\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.744638 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.744596 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:40.898877 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:40.898771 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 06 15:36:40.902845 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:36:40.902817 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96d5d40_01bd_4492_bd78_9219c3e1cf13.slice/crio-784408d443f32c09315967962129ae37d1c7e5cec35624aff9f8d446d90bbc5e WatchSource:0}: Error finding container 784408d443f32c09315967962129ae37d1c7e5cec35624aff9f8d446d90bbc5e: Status 404 returned error can't find the container with id 784408d443f32c09315967962129ae37d1c7e5cec35624aff9f8d446d90bbc5e Jul 06 15:36:41.363479 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:41.363434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"5f7f7e9523216336ad88493a75e6854cf308b6529ee4f96c821eea6dd8208aae"} Jul 06 15:36:41.363930 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:41.363512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"784408d443f32c09315967962129ae37d1c7e5cec35624aff9f8d446d90bbc5e"} Jul 06 15:36:41.605832 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:41.605799 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b" path="/var/lib/kubelet/pods/ffb42d9f-9c95-4f62-b91e-1a3d7c216a2b/volumes" Jul 06 15:36:42.369902 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:42.369861 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" event={"ID":"f29da444-c54c-44aa-876b-f02a6c5f05d9","Type":"ContainerStarted","Data":"4e2c9c8570aa253f0c39a0f4dcd3b88a396d7ab35b7f43a15e8d3575aa8c7684"} Jul 06 15:36:42.370361 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:42.369937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" event={"ID":"f29da444-c54c-44aa-876b-f02a6c5f05d9","Type":"ContainerStarted","Data":"5f47b25182a4573aa3a838c8e0d646d8058e2fb8b3a7f82fcf43c339f3c204c2"} Jul 06 15:36:42.370361 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:42.369967 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" event={"ID":"f29da444-c54c-44aa-876b-f02a6c5f05d9","Type":"ContainerStarted","Data":"0a0d1a28939d73b1af5d27cfe6b1e28885dbc79c8f4a8ed61d2d53319c4fa862"} Jul 06 15:36:42.371403 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:42.371376 2575 generic.go:356] "Generic (PLEG): container finished" podID="b96d5d40-01bd-4492-bd78-9219c3e1cf13" containerID="5f7f7e9523216336ad88493a75e6854cf308b6529ee4f96c821eea6dd8208aae" exitCode=0 Jul 06 15:36:42.371526 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:42.371432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerDied","Data":"5f7f7e9523216336ad88493a75e6854cf308b6529ee4f96c821eea6dd8208aae"} Jul 06 15:36:42.394678 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:42.394621 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-64dfc4c9-5bjwk" podStartSLOduration=2.3223385309999998 podStartE2EDuration="4.394598617s" totalImagesPullingTime="2.072260086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:36:42.393815048 +0000 UTC m=+171.328565094" watchObservedRunningTime="2026-07-06 15:36:42.394598617 +0000 UTC m=+171.329348667" Jul 06 15:36:43.112726 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.112690 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:43.213357 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.213322 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:43.379360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.379271 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"dc3e36bbc21fee473df1022290f897136d035a52367891af0701a2178cc15864"} Jul 06 15:36:43.379360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.379349 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"6e3cf3017195e29791e88d90e70b3313ce86c881b42056e66ce7139fb905702f"} Jul 06 15:36:43.379740 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.379379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"ce35ca3a3829431afc4a57c2fbabbbee8bfa7c64961929291d6e498d78b5e5e4"} Jul 06 15:36:43.379740 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.379403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"975c45b313df1fb3fdd01eee668f2749dc50120979c8084130a47eef416dd08b"} Jul 06 15:36:43.379740 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.379436 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"14878a40daef976abc7c78cd7846a247af2d06b63d50f5d46a0118616031b2c2"} Jul 06 15:36:43.379740 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.379460 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b96d5d40-01bd-4492-bd78-9219c3e1cf13","Type":"ContainerStarted","Data":"ff238c971fad888e4f17f736269c3a09953bb9728fd4ff749d2c6780faa5318e"} Jul 06 15:36:43.413069 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:43.413012 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.393701698 podStartE2EDuration="3.412995192s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.293494ms" isStatefulPod=false podCreationTimestamp="2026-07-06 15:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:36:43.410946375 +0000 UTC m=+172.345696442" watchObservedRunningTime="2026-07-06 15:36:43.412995192 +0000 UTC m=+172.347745238" Jul 06 15:36:45.745117 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:45.745075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:36:54.036862 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:54.036826 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:54.141736 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:54.141704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:36:54.216004 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:36:54.215971 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:13.461022 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:13.460983 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:13.525441 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:13.525410 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:29.386125 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:29.386091 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:40.745782 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:40.745749 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:37:40.761476 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:40.761451 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:37:41.609078 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:41.609053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 06 15:37:49.090861 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:49.090828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:49.165821 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:49.165793 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:58.743751 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:58.743715 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:37:58.845004 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:37:58.844973 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:01.140923 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:01.140890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:01.228848 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:01.228820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:06.017827 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:06.017787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:06.118559 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:06.118528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:09.345811 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:09.345772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:09.445796 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:09.445750 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:21.189954 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:21.189867 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:38:51.435257 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:38:51.435227 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 15:39:15.476537 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:39:15.476500 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:16.618185 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:39:16.618115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:33.832673 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:39:33.832641 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:39:33.927622 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:39:33.927583 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:32.487149 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:32.487102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:32.825422 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:32.825390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:36.322031 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:36.321993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:36.426089 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:36.426061 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:40.214055 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:40.214018 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:40.323885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:40.323850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:40.596927 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:40.596895 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:41.011505 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:41.011421 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:42.322467 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:42.322429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:42.978695 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:42.978660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:50.915082 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:50.915012 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:52.669604 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:52.669567 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:52.809565 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:52.809532 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:54.313381 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:54.313346 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:54.392098 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:54.392050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:59.762264 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:59.762232 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:40:59.861445 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:40:59.861404 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:01.965201 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:01.965144 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:17.644300 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:17.644221 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:17.764212 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:17.764180 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:18.236097 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.236062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:18.746580 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.746540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:18.866172 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.866128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:18.893128 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.893103 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99"] Jul 06 15:41:18.896419 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.896401 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:18.899518 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.899502 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 15:41:18.899611 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.899523 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dj8mj\"" Jul 06 15:41:18.899611 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.899504 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 15:41:18.899958 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.899940 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 15:41:18.902108 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.902090 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 15:41:18.917016 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.916996 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99"] Jul 06 15:41:18.929804 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.929780 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtm6\" (UniqueName: \"kubernetes.io/projected/3951674d-a6dd-4297-a41f-41b48426e583-kube-api-access-kxtm6\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:18.929916 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.929824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3951674d-a6dd-4297-a41f-41b48426e583-tmp\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:18.929916 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.929868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3951674d-a6dd-4297-a41f-41b48426e583-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:18.929916 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.929895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3951674d-a6dd-4297-a41f-41b48426e583-webhook-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:18.930036 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:18.929985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3951674d-a6dd-4297-a41f-41b48426e583-manifests\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.031013 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.030989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3951674d-a6dd-4297-a41f-41b48426e583-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.031214 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.031021 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3951674d-a6dd-4297-a41f-41b48426e583-webhook-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.031214 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.031050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3951674d-a6dd-4297-a41f-41b48426e583-manifests\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.031214 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.031083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtm6\" (UniqueName: \"kubernetes.io/projected/3951674d-a6dd-4297-a41f-41b48426e583-kube-api-access-kxtm6\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.031386 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.031208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3951674d-a6dd-4297-a41f-41b48426e583-tmp\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.031489 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.031466 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3951674d-a6dd-4297-a41f-41b48426e583-manifests\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.033596 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.033577 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3951674d-a6dd-4297-a41f-41b48426e583-tmp\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.033707 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.033657 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3951674d-a6dd-4297-a41f-41b48426e583-apiservice-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.033707 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.033661 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3951674d-a6dd-4297-a41f-41b48426e583-webhook-cert\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.043295 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.043276 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtm6\" (UniqueName: \"kubernetes.io/projected/3951674d-a6dd-4297-a41f-41b48426e583-kube-api-access-kxtm6\") pod \"opendatahub-operator-controller-manager-749df7788b-f6b99\" (UID: \"3951674d-a6dd-4297-a41f-41b48426e583\") " pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.056602 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.056580 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:19.157137 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.157101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:19.206118 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.206085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:19.260885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.260859 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:19.363630 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.363554 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99"] Jul 06 15:41:19.364636 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.364604 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:19.368074 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:41:19.368050 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3951674d_a6dd_4297_a41f_41b48426e583.slice/crio-1d84db964b2f1ea82c9b23953a91b182f5c97d620bf8c7b44d4d159fb16da1ec WatchSource:0}: Error finding container 1d84db964b2f1ea82c9b23953a91b182f5c97d620bf8c7b44d4d159fb16da1ec: Status 404 returned error can't find the container with id 1d84db964b2f1ea82c9b23953a91b182f5c97d620bf8c7b44d4d159fb16da1ec Jul 06 15:41:19.369667 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.369651 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:41:19.383539 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.383515 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" event={"ID":"3951674d-a6dd-4297-a41f-41b48426e583","Type":"ContainerStarted","Data":"1d84db964b2f1ea82c9b23953a91b182f5c97d620bf8c7b44d4d159fb16da1ec"} Jul 06 15:41:19.478124 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.478087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:19.586924 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.586835 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:19.695855 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:19.695821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:20.345844 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:20.345741 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:20.455114 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:20.455077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:23.402151 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:23.402117 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" event={"ID":"3951674d-a6dd-4297-a41f-41b48426e583","Type":"ContainerStarted","Data":"4c52d094c45af0ad0ee81dfbb8a094d6242f47751e2352b398d941073bbe9673"} Jul 06 15:41:24.407284 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:24.407242 2575 generic.go:356] "Generic (PLEG): container finished" podID="3951674d-a6dd-4297-a41f-41b48426e583" containerID="4c52d094c45af0ad0ee81dfbb8a094d6242f47751e2352b398d941073bbe9673" exitCode=0 Jul 06 15:41:24.407707 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:24.407338 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" event={"ID":"3951674d-a6dd-4297-a41f-41b48426e583","Type":"ContainerDied","Data":"4c52d094c45af0ad0ee81dfbb8a094d6242f47751e2352b398d941073bbe9673"} Jul 06 15:41:25.413014 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:25.412982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" event={"ID":"3951674d-a6dd-4297-a41f-41b48426e583","Type":"ContainerStarted","Data":"d891a73c0246602f715f4a48beaa91d1d12fa1281c6af55d35c3307e4260801a"} Jul 06 15:41:25.413463 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:25.413116 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:25.441811 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:25.441756 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" podStartSLOduration=3.063572853 podStartE2EDuration="7.44174147s" totalImagesPullingTime="3.313512551s" totalInitContainerRuntime="1.064656066s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:25.441361658 +0000 UTC m=+454.376111718" watchObservedRunningTime="2026-07-06 15:41:25.44174147 +0000 UTC m=+454.376491516" Jul 06 15:41:36.419931 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:36.419903 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-749df7788b-f6b99" Jul 06 15:41:38.524779 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:38.524740 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:38.692800 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:38.692764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:39.480446 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:39.480408 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:39.708484 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:39.708448 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:39.892128 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:39.892093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:40.041871 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:40.041841 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:40.223281 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:40.223189 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:40.912693 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:40.912664 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:41.091097 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:41.091055 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:41.878052 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:41.878020 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:41.975005 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:41.974970 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:43.436030 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:43.435996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:46.126775 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.126737 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw"] Jul 06 15:41:46.130113 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.130094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.135690 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.135650 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lz2bv\"" Jul 06 15:41:46.136179 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.136148 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 15:41:46.137230 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.137208 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 15:41:46.137868 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.137854 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 15:41:46.138026 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.138009 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 15:41:46.138177 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.138149 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 15:41:46.161029 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.160990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw"] Jul 06 15:41:46.256953 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.256903 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f76fae5-6ded-479a-a984-792b7d5eb430-cert\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.256953 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.256955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f76fae5-6ded-479a-a984-792b7d5eb430-metrics-cert\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.257189 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.256982 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f76fae5-6ded-479a-a984-792b7d5eb430-manager-config\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.257189 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.257022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frddv\" (UniqueName: \"kubernetes.io/projected/9f76fae5-6ded-479a-a984-792b7d5eb430-kube-api-access-frddv\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.357447 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.357396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f76fae5-6ded-479a-a984-792b7d5eb430-cert\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.357447 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.357454 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f76fae5-6ded-479a-a984-792b7d5eb430-metrics-cert\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.357680 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.357491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f76fae5-6ded-479a-a984-792b7d5eb430-manager-config\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.357680 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.357525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frddv\" (UniqueName: \"kubernetes.io/projected/9f76fae5-6ded-479a-a984-792b7d5eb430-kube-api-access-frddv\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.358143 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.358121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9f76fae5-6ded-479a-a984-792b7d5eb430-manager-config\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.360038 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.360016 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9f76fae5-6ded-479a-a984-792b7d5eb430-metrics-cert\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.360108 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.360068 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9f76fae5-6ded-479a-a984-792b7d5eb430-cert\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.370454 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.370435 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frddv\" (UniqueName: \"kubernetes.io/projected/9f76fae5-6ded-479a-a984-792b7d5eb430-kube-api-access-frddv\") pod \"lws-controller-manager-544dccfdb4-fzxlw\" (UID: \"9f76fae5-6ded-479a-a984-792b7d5eb430\") " pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.439511 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.439437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:46.582020 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:46.581978 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw"] Jul 06 15:41:46.584606 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:41:46.584578 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f76fae5_6ded_479a_a984_792b7d5eb430.slice/crio-ad2dbdf0255cde2e7c58b169bb40207198e37a70191cc7a1d8d102a33cca29db WatchSource:0}: Error finding container ad2dbdf0255cde2e7c58b169bb40207198e37a70191cc7a1d8d102a33cca29db: Status 404 returned error can't find the container with id ad2dbdf0255cde2e7c58b169bb40207198e37a70191cc7a1d8d102a33cca29db Jul 06 15:41:47.161392 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:47.161357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:47.339283 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:47.339248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:47.496712 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:47.496629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" event={"ID":"9f76fae5-6ded-479a-a984-792b7d5eb430","Type":"ContainerStarted","Data":"ad2dbdf0255cde2e7c58b169bb40207198e37a70191cc7a1d8d102a33cca29db"} Jul 06 15:41:48.462348 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.462309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:48.685031 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.684999 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg"] Jul 06 15:41:48.688292 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.688271 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.692396 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.692371 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 15:41:48.692609 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.692590 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 15:41:48.692702 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.692618 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-tgzdx\"" Jul 06 15:41:48.693399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.693383 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 15:41:48.693399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.693395 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 15:41:48.693530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.693422 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 15:41:48.693668 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.693653 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 15:41:48.706027 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.706002 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg"] Jul 06 15:41:48.781438 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.781409 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.781438 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.781442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flhj\" (UniqueName: \"kubernetes.io/projected/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-kube-api-access-8flhj\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.781629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.781478 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.781629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.781536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.781629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.781571 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.781629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.781591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.781629 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.781613 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.882155 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.882112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.882357 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.882206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.882357 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.882243 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.882357 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.882281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.882357 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.882316 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.882357 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.882342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flhj\" (UniqueName: \"kubernetes.io/projected/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-kube-api-access-8flhj\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.882607 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.882399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.883514 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.883288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.885529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.885262 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.885529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.885419 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.885529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.885471 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.885763 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.885606 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.892626 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.892607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flhj\" (UniqueName: \"kubernetes.io/projected/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-kube-api-access-8flhj\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.918256 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.918232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5t4zg\" (UID: \"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:48.999130 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:48.999093 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:49.059791 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.059708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:49.180847 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.180750 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:49.287715 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.287684 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg"] Jul 06 15:41:49.289957 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:41:49.289929 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7a9ceb_0fbf_41f1_870e_f7b0c19361c9.slice/crio-c0e5d0925cdf30cd5b01894809ce45f8f9dbbd0413312a89800feb28efc67804 WatchSource:0}: Error finding container c0e5d0925cdf30cd5b01894809ce45f8f9dbbd0413312a89800feb28efc67804: Status 404 returned error can't find the container with id c0e5d0925cdf30cd5b01894809ce45f8f9dbbd0413312a89800feb28efc67804 Jul 06 15:41:49.507455 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.507371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" event={"ID":"9f76fae5-6ded-479a-a984-792b7d5eb430","Type":"ContainerStarted","Data":"9dc32d3e67168c6c2498c414ac59de691a07f007ea718434d2c1021eff03d137"} Jul 06 15:41:49.507455 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.507436 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:41:49.508569 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.508548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" event={"ID":"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9","Type":"ContainerStarted","Data":"c0e5d0925cdf30cd5b01894809ce45f8f9dbbd0413312a89800feb28efc67804"} Jul 06 15:41:49.524963 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.524938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:49.559814 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:49.559762 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" podStartSLOduration=0.938572422 podStartE2EDuration="3.559747961s" totalImagesPullingTime="2.621175539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:49.558940706 +0000 UTC m=+478.493690754" watchObservedRunningTime="2026-07-06 15:41:49.559747961 +0000 UTC m=+478.494498008" Jul 06 15:41:52.050905 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:52.050854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:52.161508 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:52.161471 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:52.279128 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:52.279097 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:52.493374 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:52.493233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:53.348540 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:53.348506 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:53.453387 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:53.453355 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:53.552515 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:53.552477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:55.002984 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:55.002932 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 15:41:55.003270 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:55.003012 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 06 15:41:55.544920 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:55.544875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" event={"ID":"0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9","Type":"ContainerStarted","Data":"a615054cf15dd1bf5a3e38ff2846531fc7a43fc27922915d3b668ea7e5382fd7"} Jul 06 15:41:55.545084 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:55.544971 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:56.552424 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:56.552397 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" Jul 06 15:41:56.592398 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:56.592339 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5t4zg" podStartSLOduration=2.881591385 podStartE2EDuration="8.592323462s" totalImagesPullingTime="5.710732077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:41:55.574000303 +0000 UTC m=+484.508750351" watchObservedRunningTime="2026-07-06 15:41:56.592323462 +0000 UTC m=+485.527073508" Jul 06 15:41:59.556250 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:59.556220 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:41:59.646155 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:41:59.646119 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:00.515058 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:42:00.515029 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-544dccfdb4-fzxlw" Jul 06 15:42:36.842448 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:42:36.842412 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:36.901900 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:42:36.901864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:40.172180 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:42:40.172138 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:40.269639 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:42:40.269601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:42:40.350651 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:42:40.350618 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:01.970926 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:01.970893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:09.321049 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:09.320992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:12.715146 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:12.715096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:13.517632 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:13.517595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:16.710438 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:16.710407 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:16.822718 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:16.822687 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:19.210738 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:19.210687 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:21.000364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:21.000318 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:22.759880 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:22.759842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:25.799635 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.799577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:25.826614 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.826572 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt"] Jul 06 15:43:25.830121 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.830096 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" Jul 06 15:43:25.832567 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.832546 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 15:43:25.833497 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.833478 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 15:43:25.833595 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.833506 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f4cx4\"" Jul 06 15:43:25.833595 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.833506 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 15:43:25.842438 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.842410 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt"] Jul 06 15:43:25.925119 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:25.925076 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzc7f\" (UniqueName: \"kubernetes.io/projected/980a6f0f-5f46-4408-814e-ff7f1999187e-kube-api-access-tzc7f\") pod \"dns-operator-controller-manager-864dd78c4c-zslmt\" (UID: \"980a6f0f-5f46-4408-814e-ff7f1999187e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" Jul 06 15:43:26.026426 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:26.026386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzc7f\" (UniqueName: \"kubernetes.io/projected/980a6f0f-5f46-4408-814e-ff7f1999187e-kube-api-access-tzc7f\") pod \"dns-operator-controller-manager-864dd78c4c-zslmt\" (UID: \"980a6f0f-5f46-4408-814e-ff7f1999187e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" Jul 06 15:43:26.040211 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:26.040182 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzc7f\" (UniqueName: \"kubernetes.io/projected/980a6f0f-5f46-4408-814e-ff7f1999187e-kube-api-access-tzc7f\") pod \"dns-operator-controller-manager-864dd78c4c-zslmt\" (UID: \"980a6f0f-5f46-4408-814e-ff7f1999187e\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" Jul 06 15:43:26.140296 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:26.140195 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" Jul 06 15:43:26.277319 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:26.277240 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt"] Jul 06 15:43:26.281116 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:43:26.281088 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980a6f0f_5f46_4408_814e_ff7f1999187e.slice/crio-8b5d1c2ea8c0777de68f507cda7bf8f10a3b047b55dca0f87ccc39506a5d8e9d WatchSource:0}: Error finding container 8b5d1c2ea8c0777de68f507cda7bf8f10a3b047b55dca0f87ccc39506a5d8e9d: Status 404 returned error can't find the container with id 8b5d1c2ea8c0777de68f507cda7bf8f10a3b047b55dca0f87ccc39506a5d8e9d Jul 06 15:43:26.933364 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:26.933327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" event={"ID":"980a6f0f-5f46-4408-814e-ff7f1999187e","Type":"ContainerStarted","Data":"8b5d1c2ea8c0777de68f507cda7bf8f10a3b047b55dca0f87ccc39506a5d8e9d"} Jul 06 15:43:28.344317 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:28.344278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:28.443138 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:28.443102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:28.943839 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:28.943755 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" event={"ID":"980a6f0f-5f46-4408-814e-ff7f1999187e","Type":"ContainerStarted","Data":"5b2a773cd701cff27017e7628f8304e5c86015ed0b8ffd237c9000412b9d48d5"} Jul 06 15:43:28.943983 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:28.943894 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" Jul 06 15:43:28.962560 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:28.962497 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" podStartSLOduration=1.554404997 podStartE2EDuration="3.962480791s" totalImagesPullingTime="2.408075794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:28.962337978 +0000 UTC m=+577.897088026" watchObservedRunningTime="2026-07-06 15:43:28.962480791 +0000 UTC m=+577.897230837" Jul 06 15:43:30.936501 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:30.936443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:31.170141 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:31.170101 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:33.318382 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:33.318349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:33.418990 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:33.418952 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:33.926529 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:33.926497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:34.002016 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:34.001986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:37.749127 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:37.749077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:37.850491 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:37.850458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:39.950365 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:39.950334 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zslmt" Jul 06 15:43:41.690317 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.690285 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw"] Jul 06 15:43:41.693770 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.693752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:41.701233 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.698531 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f6w8p\"" Jul 06 15:43:41.713796 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.713764 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw"] Jul 06 15:43:41.773608 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.773574 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw"] Jul 06 15:43:41.773907 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:41.773880 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-fmxc4], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" podUID="fb5065b9-ee9c-43fd-be54-fab61371a3a9" Jul 06 15:43:41.783394 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.783357 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw"] Jul 06 15:43:41.868976 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.868946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxc4\" (UniqueName: \"kubernetes.io/projected/fb5065b9-ee9c-43fd-be54-fab61371a3a9-kube-api-access-fmxc4\") pod \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" (UID: \"fb5065b9-ee9c-43fd-be54-fab61371a3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:41.869126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.869005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5065b9-ee9c-43fd-be54-fab61371a3a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" (UID: \"fb5065b9-ee9c-43fd-be54-fab61371a3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:41.869548 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.869523 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75"] Jul 06 15:43:41.872798 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.872778 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:41.933522 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.933489 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75"] Jul 06 15:43:41.970203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.970081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66cb61b0-4261-4182-9885-c4e50757826d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-68b75\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:41.970203 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.970137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wksqw\" (UniqueName: \"kubernetes.io/projected/66cb61b0-4261-4182-9885-c4e50757826d-kube-api-access-wksqw\") pod \"kuadrant-operator-controller-manager-86878c769f-68b75\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:41.970448 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.970237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxc4\" (UniqueName: \"kubernetes.io/projected/fb5065b9-ee9c-43fd-be54-fab61371a3a9-kube-api-access-fmxc4\") pod \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" (UID: \"fb5065b9-ee9c-43fd-be54-fab61371a3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:41.970448 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.970332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5065b9-ee9c-43fd-be54-fab61371a3a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" (UID: \"fb5065b9-ee9c-43fd-be54-fab61371a3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:41.970714 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.970694 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5065b9-ee9c-43fd-be54-fab61371a3a9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" (UID: \"fb5065b9-ee9c-43fd-be54-fab61371a3a9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:41.976117 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:41.976097 2575 projected.go:196] Error preparing data for projected volume kube-api-access-fmxc4 for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-r4qvw" not found Jul 06 15:43:41.976244 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:41.976176 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb5065b9-ee9c-43fd-be54-fab61371a3a9-kube-api-access-fmxc4 podName:fb5065b9-ee9c-43fd-be54-fab61371a3a9 nodeName:}" failed. No retries permitted until 2026-07-06 15:43:42.476145963 +0000 UTC m=+591.410895987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fmxc4" (UniqueName: "kubernetes.io/projected/fb5065b9-ee9c-43fd-be54-fab61371a3a9-kube-api-access-fmxc4") pod "kuadrant-operator-controller-manager-86878c769f-r4qvw" (UID: "fb5065b9-ee9c-43fd-be54-fab61371a3a9") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-r4qvw" not found Jul 06 15:43:41.997065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:41.997030 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:41.999779 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:41.999746 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" is forbidden: User \"system:node:ip-10-0-131-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-241.ec2.internal' and this object" podUID="fb5065b9-ee9c-43fd-be54-fab61371a3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:42.003540 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.003519 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:42.005671 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:42.005645 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" is forbidden: User \"system:node:ip-10-0-131-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-241.ec2.internal' and this object" podUID="fb5065b9-ee9c-43fd-be54-fab61371a3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:42.071188 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.071130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksqw\" (UniqueName: \"kubernetes.io/projected/66cb61b0-4261-4182-9885-c4e50757826d-kube-api-access-wksqw\") pod \"kuadrant-operator-controller-manager-86878c769f-68b75\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:42.071351 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.071289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66cb61b0-4261-4182-9885-c4e50757826d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-68b75\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:42.071664 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.071645 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66cb61b0-4261-4182-9885-c4e50757826d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-68b75\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:42.086459 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.086433 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksqw\" (UniqueName: \"kubernetes.io/projected/66cb61b0-4261-4182-9885-c4e50757826d-kube-api-access-wksqw\") pod \"kuadrant-operator-controller-manager-86878c769f-68b75\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:42.171990 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.171955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5065b9-ee9c-43fd-be54-fab61371a3a9-extensions-socket-volume\") pod \"fb5065b9-ee9c-43fd-be54-fab61371a3a9\" (UID: \"fb5065b9-ee9c-43fd-be54-fab61371a3a9\") " Jul 06 15:43:42.172155 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.172100 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmxc4\" (UniqueName: \"kubernetes.io/projected/fb5065b9-ee9c-43fd-be54-fab61371a3a9-kube-api-access-fmxc4\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:43:42.172238 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.172215 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb5065b9-ee9c-43fd-be54-fab61371a3a9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fb5065b9-ee9c-43fd-be54-fab61371a3a9" (UID: "fb5065b9-ee9c-43fd-be54-fab61371a3a9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:43:42.183069 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.183040 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:42.272766 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.272736 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5065b9-ee9c-43fd-be54-fab61371a3a9-extensions-socket-volume\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:43:42.327097 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:42.327063 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75"] Jul 06 15:43:42.330475 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:43:42.330445 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cb61b0_4261_4182_9885_c4e50757826d.slice/crio-cbf00af9ee298d55d6e8d9af18bff77e281961c5354769d2f680ca545c3741bd WatchSource:0}: Error finding container cbf00af9ee298d55d6e8d9af18bff77e281961c5354769d2f680ca545c3741bd: Status 404 returned error can't find the container with id cbf00af9ee298d55d6e8d9af18bff77e281961c5354769d2f680ca545c3741bd Jul 06 15:43:43.002753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:43.002717 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" event={"ID":"66cb61b0-4261-4182-9885-c4e50757826d","Type":"ContainerStarted","Data":"cbf00af9ee298d55d6e8d9af18bff77e281961c5354769d2f680ca545c3741bd"} Jul 06 15:43:43.003194 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:43.002762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:43.005210 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:43.005155 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" is forbidden: User \"system:node:ip-10-0-131-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-241.ec2.internal' and this object" podUID="fb5065b9-ee9c-43fd-be54-fab61371a3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:43.013713 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:43.013663 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-r4qvw\" is forbidden: User \"system:node:ip-10-0-131-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-241.ec2.internal' and this object" podUID="fb5065b9-ee9c-43fd-be54-fab61371a3a9" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r4qvw" Jul 06 15:43:43.032141 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:43:43.032107 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5065b9_ee9c_43fd_be54_fab61371a3a9.slice\": RecentStats: unable to find data in memory cache]" Jul 06 15:43:43.604728 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:43.604689 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb5065b9-ee9c-43fd-be54-fab61371a3a9" path="/var/lib/kubelet/pods/fb5065b9-ee9c-43fd-be54-fab61371a3a9/volumes" Jul 06 15:43:43.647906 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:43.647847 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:43.781239 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:43.781200 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:45.863065 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:45.863024 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:48.029563 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:48.029528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" event={"ID":"66cb61b0-4261-4182-9885-c4e50757826d","Type":"ContainerStarted","Data":"eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8"} Jul 06 15:43:48.029947 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:48.029656 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:43:48.116179 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:48.116113 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" podStartSLOduration=1.9497794160000002 podStartE2EDuration="7.116096309s" totalImagesPullingTime="5.166316893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:43:48.11455402 +0000 UTC m=+597.049304080" watchObservedRunningTime="2026-07-06 15:43:48.116096309 +0000 UTC m=+597.050846355" Jul 06 15:43:48.311878 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:48.311845 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:48.423683 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:48.423644 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:53.165789 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:53.165693 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:53.248371 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:53.248334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:53.374527 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:53.374498 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:53.483851 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:53.483773 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:56.406360 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:56.406325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:56.519599 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:56.519570 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:43:59.035934 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:43:59.035904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:44:02.877541 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:02.877494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:02.957268 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:02.957236 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:06.317090 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:06.317051 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:15.766557 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:15.766522 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:16.017025 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:16.016942 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:18.225804 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.225768 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:18.689931 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.689893 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:44:18.734635 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.734590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:44:18.734635 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.734635 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:44:18.734890 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.734738 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:18.738919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.738891 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 15:44:18.738919 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.738915 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jk25k\"" Jul 06 15:44:18.894871 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.894835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db951259-31f9-4dce-b5ed-595094f981a3-config-file\") pod \"limitador-limitador-54d6d988d5-gjbhb\" (UID: \"db951259-31f9-4dce-b5ed-595094f981a3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:18.895054 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.894888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwsq\" (UniqueName: \"kubernetes.io/projected/db951259-31f9-4dce-b5ed-595094f981a3-kube-api-access-4lwsq\") pod \"limitador-limitador-54d6d988d5-gjbhb\" (UID: \"db951259-31f9-4dce-b5ed-595094f981a3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:18.995659 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.995563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db951259-31f9-4dce-b5ed-595094f981a3-config-file\") pod \"limitador-limitador-54d6d988d5-gjbhb\" (UID: \"db951259-31f9-4dce-b5ed-595094f981a3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:18.995659 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.995615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwsq\" (UniqueName: \"kubernetes.io/projected/db951259-31f9-4dce-b5ed-595094f981a3-kube-api-access-4lwsq\") pod \"limitador-limitador-54d6d988d5-gjbhb\" (UID: \"db951259-31f9-4dce-b5ed-595094f981a3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:18.996333 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:18.996311 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db951259-31f9-4dce-b5ed-595094f981a3-config-file\") pod \"limitador-limitador-54d6d988d5-gjbhb\" (UID: \"db951259-31f9-4dce-b5ed-595094f981a3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:19.005118 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:19.005092 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwsq\" (UniqueName: \"kubernetes.io/projected/db951259-31f9-4dce-b5ed-595094f981a3-kube-api-access-4lwsq\") pod \"limitador-limitador-54d6d988d5-gjbhb\" (UID: \"db951259-31f9-4dce-b5ed-595094f981a3\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:19.045638 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:19.045604 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:19.212786 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:19.212705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:44:19.215521 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:44:19.215488 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb951259_31f9_4dce_b5ed_595094f981a3.slice/crio-64a7550864eed6c9784ca5b44678501e93306f8d9dc1897acda91074ea2a8c57 WatchSource:0}: Error finding container 64a7550864eed6c9784ca5b44678501e93306f8d9dc1897acda91074ea2a8c57: Status 404 returned error can't find the container with id 64a7550864eed6c9784ca5b44678501e93306f8d9dc1897acda91074ea2a8c57 Jul 06 15:44:19.447380 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:19.447346 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:20.167032 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:20.166919 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" event={"ID":"db951259-31f9-4dce-b5ed-595094f981a3","Type":"ContainerStarted","Data":"64a7550864eed6c9784ca5b44678501e93306f8d9dc1897acda91074ea2a8c57"} Jul 06 15:44:22.180305 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:22.180269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" event={"ID":"db951259-31f9-4dce-b5ed-595094f981a3","Type":"ContainerStarted","Data":"b0be5c5b6100ef3c2530a965158c828b5b038f8f728e93907ef5a074eba14137"} Jul 06 15:44:22.180726 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:22.180353 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:22.205513 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:22.205468 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" podStartSLOduration=1.822432451 podStartE2EDuration="4.205453389s" totalImagesPullingTime="2.383020938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:44:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:44:22.204156067 +0000 UTC m=+631.138906114" watchObservedRunningTime="2026-07-06 15:44:22.205453389 +0000 UTC m=+631.140203466" Jul 06 15:44:27.495469 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:27.495426 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:27.595870 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:27.595832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:33.185795 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:33.185765 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gjbhb" Jul 06 15:44:39.870658 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:39.870624 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:52.562492 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:52.562454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:52.704335 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:52.704299 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:58.965328 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:58.965296 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:59.157953 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:59.157917 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:44:59.263433 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:44:59.263350 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:02.157691 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:02.157655 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:02.251474 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:02.251439 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:33.560897 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:33.560860 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:33.988697 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:33.988616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:34.153137 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:34.153096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:34.465305 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:34.465270 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:34.569713 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:34.569675 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:34.709100 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:34.709066 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:45:45.008095 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:45.008020 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:45.104747 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:45.104713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:58.332638 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:58.332595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:45:58.753755 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:45:58.753676 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:46:22.691754 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:22.691708 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:23.342816 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:23.342781 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:46:27.761496 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:27.761449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:28.635603 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:28.635567 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:46:37.569876 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:37.569841 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:39.043563 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:39.043526 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:46:41.947289 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.947247 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8"] Jul 06 15:46:41.950920 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.950901 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:41.953977 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.953953 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 15:46:41.953977 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.953976 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 15:46:41.954264 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.954245 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 15:46:41.954725 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.954708 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cmsdq\"" Jul 06 15:46:41.961513 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.961493 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8"] Jul 06 15:46:41.979014 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:41.978980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:46:42.142650 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.142606 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.142826 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.142675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.142826 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.142706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqx5x\" (UniqueName: \"kubernetes.io/projected/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-kube-api-access-bqx5x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.142826 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.142768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.142826 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.142800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.142958 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.142869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.243720 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.243636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.243720 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.243679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.243955 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.243722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.243955 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.243763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.243955 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.243816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.243955 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.243852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqx5x\" (UniqueName: \"kubernetes.io/projected/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-kube-api-access-bqx5x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.244236 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.244097 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.244236 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.244149 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.244351 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.244332 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.246227 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.246203 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.246493 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.246471 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.262801 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.262769 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqx5x\" (UniqueName: \"kubernetes.io/projected/9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad-kube-api-access-bqx5x\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b7js8\" (UID: \"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.562181 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.562139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:42.711691 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.711645 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8"] Jul 06 15:46:42.716547 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:46:42.716518 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aac0d1b_ad7a_40bd_9405_4baee1e9b1ad.slice/crio-368dd36b5d5a81727e71a3df57862ee2155d52f6be0c17ea85bf9614e4d99eb1 WatchSource:0}: Error finding container 368dd36b5d5a81727e71a3df57862ee2155d52f6be0c17ea85bf9614e4d99eb1: Status 404 returned error can't find the container with id 368dd36b5d5a81727e71a3df57862ee2155d52f6be0c17ea85bf9614e4d99eb1 Jul 06 15:46:42.717773 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.717751 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 15:46:42.790683 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:42.790641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" event={"ID":"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad","Type":"ContainerStarted","Data":"368dd36b5d5a81727e71a3df57862ee2155d52f6be0c17ea85bf9614e4d99eb1"} Jul 06 15:46:44.248415 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:44.248323 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:46:44.804301 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:44.804269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" event={"ID":"9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad","Type":"ContainerStarted","Data":"7e771777ea6569f35034d710eb40f5f7f5db539e5b091de6fdb48c9a70ae57c0"} Jul 06 15:46:44.827750 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:44.827694 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" podStartSLOduration=2.620804631 podStartE2EDuration="3.827678658s" totalImagesPullingTime="1.206874027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:46:44.826833431 +0000 UTC m=+773.761583490" watchObservedRunningTime="2026-07-06 15:46:44.827678658 +0000 UTC m=+773.762428683" Jul 06 15:46:52.562399 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:52.562359 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:52.562803 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:52.562450 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:52.575025 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:52.574998 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:46:52.849405 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:46:52.849312 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b7js8" Jul 06 15:47:06.275247 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:47:06.275205 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:09.038488 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:47:09.038451 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:47:24.398941 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:47:24.398857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:24.460710 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:47:24.460673 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:37.274066 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:47:37.274029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:47:37.341275 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:47:37.341193 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:11.858684 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:11.858638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:11.948849 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:11.948815 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:13.049429 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:13.049396 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:13.158263 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:13.158233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:41.951578 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:41.951500 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:48:52.347217 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:52.347179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:48:55.987379 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:55.987329 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:48:56.049133 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:48:56.049097 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:04.142180 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:04.142127 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:49:10.310157 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:10.310118 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:10.377128 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:10.377093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:13.143765 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:13.143727 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:49:23.765739 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:23.765701 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:49:26.057864 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:26.057831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:27.476810 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:27.476774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:49:53.053640 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:49:53.053591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:50:04.853829 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:04.853790 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:50:05.870961 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:05.870925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:10.972625 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:10.972586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:11.053377 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:11.053340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:12.959930 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:12.959857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:17.612421 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:17.612386 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:17.681350 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:17.681308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:17.770920 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:17.770886 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:17.872016 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:17.871922 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:17.974016 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:17.973976 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:24.264955 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:24.264919 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:24.373845 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:24.373805 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:35.179816 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:35.179784 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:35.279510 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:35.279480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:50:40.844626 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:40.844587 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:50:49.139329 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:49.139290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:50:57.666509 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:50:57.666471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:51:04.444126 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:51:04.444090 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:51:06.221246 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:51:06.221212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:51:14.840218 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:51:14.840174 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:51:24.051579 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:51:24.051541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:51:35.767022 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:51:35.766990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:51:45.555672 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:51:45.555591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:52:07.430199 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:52:07.430146 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:52:17.170502 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:52:17.170458 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:52:46.749233 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:52:46.749197 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:52:59.262606 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:52:59.262569 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:53:13.566654 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:53:13.566562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:53:24.189531 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:53:24.189494 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:53:34.061879 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:53:34.061837 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:53:43.435814 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:53:43.435782 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:53:52.240598 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:53:52.240560 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:54:19.032072 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:19.032031 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75"] Jul 06 15:54:19.032568 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:19.032334 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" podUID="66cb61b0-4261-4182-9885-c4e50757826d" containerName="manager" containerID="cri-o://eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8" gracePeriod=10 Jul 06 15:54:21.786076 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.786049 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:54:21.807095 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.807070 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66cb61b0-4261-4182-9885-c4e50757826d-extensions-socket-volume\") pod \"66cb61b0-4261-4182-9885-c4e50757826d\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " Jul 06 15:54:21.807306 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.807132 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wksqw\" (UniqueName: \"kubernetes.io/projected/66cb61b0-4261-4182-9885-c4e50757826d-kube-api-access-wksqw\") pod \"66cb61b0-4261-4182-9885-c4e50757826d\" (UID: \"66cb61b0-4261-4182-9885-c4e50757826d\") " Jul 06 15:54:21.807466 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.807443 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cb61b0-4261-4182-9885-c4e50757826d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "66cb61b0-4261-4182-9885-c4e50757826d" (UID: "66cb61b0-4261-4182-9885-c4e50757826d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 15:54:21.809384 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.809355 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cb61b0-4261-4182-9885-c4e50757826d-kube-api-access-wksqw" (OuterVolumeSpecName: "kube-api-access-wksqw") pod "66cb61b0-4261-4182-9885-c4e50757826d" (UID: "66cb61b0-4261-4182-9885-c4e50757826d"). InnerVolumeSpecName "kube-api-access-wksqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 15:54:21.829852 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.829818 2575 generic.go:356] "Generic (PLEG): container finished" podID="66cb61b0-4261-4182-9885-c4e50757826d" containerID="eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8" exitCode=0 Jul 06 15:54:21.830008 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.829875 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" Jul 06 15:54:21.830008 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.829914 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" event={"ID":"66cb61b0-4261-4182-9885-c4e50757826d","Type":"ContainerDied","Data":"eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8"} Jul 06 15:54:21.830008 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.829965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75" event={"ID":"66cb61b0-4261-4182-9885-c4e50757826d","Type":"ContainerDied","Data":"cbf00af9ee298d55d6e8d9af18bff77e281961c5354769d2f680ca545c3741bd"} Jul 06 15:54:21.830008 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.829992 2575 scope.go:117] "RemoveContainer" containerID="eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8" Jul 06 15:54:21.840128 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.840106 2575 scope.go:117] "RemoveContainer" containerID="eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8" Jul 06 15:54:21.840663 ip-10-0-131-241 kubenswrapper[2575]: E0706 15:54:21.840573 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8\": container with ID starting with eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8 not found: ID does not exist" containerID="eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8" Jul 06 15:54:21.840663 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.840617 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8"} err="failed to get container status \"eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8\": rpc error: code = NotFound desc = could not find container \"eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8\": container with ID starting with eb3462211476c1acc99dc882df8c8b29943aad8b8d7fb30416b8720b6bc987e8 not found: ID does not exist" Jul 06 15:54:21.855855 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.855832 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75"] Jul 06 15:54:21.860646 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.860625 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-68b75"] Jul 06 15:54:21.908672 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.908580 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66cb61b0-4261-4182-9885-c4e50757826d-extensions-socket-volume\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:54:21.908672 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:21.908613 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wksqw\" (UniqueName: \"kubernetes.io/projected/66cb61b0-4261-4182-9885-c4e50757826d-kube-api-access-wksqw\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 15:54:23.603060 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:54:23.603025 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66cb61b0-4261-4182-9885-c4e50757826d" path="/var/lib/kubelet/pods/66cb61b0-4261-4182-9885-c4e50757826d/volumes" Jul 06 15:55:20.286901 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.286864 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs"] Jul 06 15:55:20.287365 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.287224 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66cb61b0-4261-4182-9885-c4e50757826d" containerName="manager" Jul 06 15:55:20.287365 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.287235 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="66cb61b0-4261-4182-9885-c4e50757826d" containerName="manager" Jul 06 15:55:20.287365 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.287290 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="66cb61b0-4261-4182-9885-c4e50757826d" containerName="manager" Jul 06 15:55:20.290196 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.290152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.295813 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.295789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f6w8p\"" Jul 06 15:55:20.309433 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.309402 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs"] Jul 06 15:55:20.338337 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.338299 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccb56\" (UniqueName: \"kubernetes.io/projected/0770cf65-eeb9-4e1e-91fa-98f9823dfdb5-kube-api-access-ccb56\") pod \"kuadrant-operator-controller-manager-86878c769f-bhqqs\" (UID: \"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.338530 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.338408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0770cf65-eeb9-4e1e-91fa-98f9823dfdb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bhqqs\" (UID: \"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.439145 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.439101 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccb56\" (UniqueName: \"kubernetes.io/projected/0770cf65-eeb9-4e1e-91fa-98f9823dfdb5-kube-api-access-ccb56\") pod \"kuadrant-operator-controller-manager-86878c769f-bhqqs\" (UID: \"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.439345 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.439209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0770cf65-eeb9-4e1e-91fa-98f9823dfdb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bhqqs\" (UID: \"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.439535 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.439519 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0770cf65-eeb9-4e1e-91fa-98f9823dfdb5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bhqqs\" (UID: \"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.452493 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.452460 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccb56\" (UniqueName: \"kubernetes.io/projected/0770cf65-eeb9-4e1e-91fa-98f9823dfdb5-kube-api-access-ccb56\") pod \"kuadrant-operator-controller-manager-86878c769f-bhqqs\" (UID: \"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.600769 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.600677 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:20.764654 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:20.764628 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs"] Jul 06 15:55:20.767697 ip-10-0-131-241 kubenswrapper[2575]: W0706 15:55:20.767671 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0770cf65_eeb9_4e1e_91fa_98f9823dfdb5.slice/crio-565ffd6b88a88c0b1ccaa77417e7ffd702481e202866a069ffd39c179f32243f WatchSource:0}: Error finding container 565ffd6b88a88c0b1ccaa77417e7ffd702481e202866a069ffd39c179f32243f: Status 404 returned error can't find the container with id 565ffd6b88a88c0b1ccaa77417e7ffd702481e202866a069ffd39c179f32243f Jul 06 15:55:21.087661 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:21.087626 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" event={"ID":"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5","Type":"ContainerStarted","Data":"40f4e3a6b738b5e63eda5829512ff8aa97dc33c288cdd1938eb4784a04f20fa2"} Jul 06 15:55:21.087852 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:21.087679 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" event={"ID":"0770cf65-eeb9-4e1e-91fa-98f9823dfdb5","Type":"ContainerStarted","Data":"565ffd6b88a88c0b1ccaa77417e7ffd702481e202866a069ffd39c179f32243f"} Jul 06 15:55:21.087852 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:21.087715 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:21.123655 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:21.123602 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" podStartSLOduration=1.12358719 podStartE2EDuration="1.12358719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:55:21.122796987 +0000 UTC m=+1290.057547056" watchObservedRunningTime="2026-07-06 15:55:21.12358719 +0000 UTC m=+1290.058337237" Jul 06 15:55:32.094588 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:32.094548 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bhqqs" Jul 06 15:55:41.151050 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:41.151010 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:55:49.951928 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:49.951896 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:55:52.596635 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:52.596588 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:55:54.751828 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:55:54.751796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:04.840984 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:04.840953 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:56:12.460313 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:12.460236 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:56:20.947348 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:20.947313 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:23.647119 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:23.647080 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:56:24.455966 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:24.455926 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:56:32.338909 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:32.338872 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:56:42.638753 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:42.638718 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:56:49.941003 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:49.940970 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:56:58.545859 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:56:58.545827 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:57:08.837822 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:57:08.837789 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:57:19.444442 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:57:19.444405 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:57:29.446014 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:57:29.445979 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:57:52.067467 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:57:52.067427 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:58:01.267622 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:58:01.267578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:58:10.444074 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:58:10.444034 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:58:21.336361 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:58:21.336324 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:58:21.457242 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:58:21.457198 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:58:36.943702 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:58:36.943668 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:58:48.042380 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:58:48.042339 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:58:56.557654 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:58:56.557621 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:59:07.957689 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:07.957636 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:59:16.550641 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:16.550560 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:59:25.573656 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:25.573616 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:59:39.132304 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:39.132264 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:59:47.947677 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:47.947642 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 15:59:49.567402 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:49.567369 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:49.666885 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:49.666849 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 15:59:57.436938 ip-10-0-131-241 kubenswrapper[2575]: I0706 15:59:57.436894 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:00:08.432079 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:00:08.432045 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:00:18.842735 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:00:18.842702 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:00:26.369458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:00:26.369422 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:00:37.036786 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:00:37.036755 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:00:48.042700 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:00:48.042613 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:00:57.936544 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:00:57.936512 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:01:00.226292 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:00.226259 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:07.257086 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:07.257052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:09.929467 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:09.929428 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:01:15.675034 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:15.675004 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:20.037263 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:20.037226 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:01:20.961250 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:20.961219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:23.436632 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:23.436595 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:23.536247 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:23.536213 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:24.397748 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:24.397713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:29.339351 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:29.339314 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:01:30.250805 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:30.250771 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:30.350606 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:30.350568 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:30.461829 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:30.461795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:30.591079 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:30.591039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:31.692642 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:31.692609 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:31.782047 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:31.782017 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:31.871404 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:31.871374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:31.957077 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:31.956998 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:32.409382 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:32.409348 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:32.506196 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:32.506150 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:34.228982 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:34.228944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:36.825239 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:36.825202 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:36.972346 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:36.972298 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:37.070495 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:37.070458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:37.536118 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:37.536088 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:38.442255 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:38.442224 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:43.364589 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:43.364552 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:43.466428 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:43.466394 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:43.571427 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:43.571382 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:43.652789 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:43.652704 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:44.787372 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:44.787340 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:44.872855 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:44.872811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:44.937849 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:44.937813 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:45.533397 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:45.533355 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:45.633847 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:45.633817 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:46.620926 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:46.620891 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:46.939273 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:46.939158 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:01:49.937516 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:49.937469 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:50.400020 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:50.399992 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:56.282325 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:56.282271 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:56.401806 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:56.401775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:57.944697 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:57.944667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:01:58.141762 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:01:58.141726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:02:02.558489 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:02.558446 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:02.660636 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:02.660601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:02.771465 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:02.771405 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:02.831596 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:02.831501 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:03.983563 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:03.983518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:04.497251 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:04.497199 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:04.591733 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:04.591673 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:04.716888 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:04.716838 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:04.824277 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:04.824240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:06.206409 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:06.206380 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:07.246034 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:07.245994 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:02:08.479527 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:08.479491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:09.168585 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:09.168544 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:09.279899 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:09.279861 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:10.634214 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:10.634154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:14.911185 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:14.911069 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:15.014254 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:15.014195 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:15.404848 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:15.404812 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:15.502804 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:15.502764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:15.606223 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:15.606191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:15.850363 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:15.850328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:02:17.120310 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:17.120274 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:23.253985 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:23.253941 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:23.763434 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:23.763393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:24.030252 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:24.030156 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:24.132292 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:24.132254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:49.991490 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:49.991445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:56.628870 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:56.628828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:56.715414 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:56.715373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:02:56.815380 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:02:56.815343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:12.795094 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:12.795046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:19.127243 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:19.127202 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:19.241862 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:19.241823 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:19.372814 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:19.372787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:29.855240 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:29.855205 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:30.700391 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:30.700359 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:36.962870 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:36.962825 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:37.067126 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:37.067049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:37.163279 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:37.163245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:37.294060 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:37.294033 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.031559 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:39.031527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.122407 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:39.122367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:39.810413 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:39.810378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:40.339923 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:40.339885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:03:45.800128 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:45.800050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:45.897706 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:45.897671 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:46.490002 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:46.489969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:49.340915 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:49.340879 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:03:52.539300 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:52.539254 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:52.640866 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:52.640834 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:52.794022 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:52.793932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:53.096153 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:53.096085 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:53.718980 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:53.718944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:55.984417 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:55.984378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:56.073819 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:56.073780 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:56.149630 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:56.149597 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:56.248347 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:56.248245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:56.925188 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:56.925129 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:03:57.009795 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:03:57.009760 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:00.501121 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:00.501088 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:00.638080 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:00.638035 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:04:01.242898 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:01.242865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:01.350120 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:01.350087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:01.705502 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:01.705470 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:08.130648 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:08.130608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:08.230950 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:08.230916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:08.577370 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:08.577334 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:08.942477 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:08.942390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:09.043298 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:09.043261 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:11.873712 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:11.873670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:04:15.447017 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:15.446985 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:15.547353 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:15.547314 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:15.718193 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:15.718077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:15.879309 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:15.879271 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:15.991361 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:15.991282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:16.105578 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:16.105541 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:16.203512 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:16.203478 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:16.303474 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:16.303442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:22.610402 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:22.610366 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:22.683793 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:22.683765 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:22.771354 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:22.771322 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:22.960135 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:22.960048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:29.587753 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:29.587713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:29.671977 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:29.671943 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:36.559945 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:36.559907 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:36.806759 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:36.806721 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:37.236358 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:37.236325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:39.935896 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:39.935862 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:40.021555 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:40.021517 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:40.096668 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:40.096625 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:40.260049 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:40.259964 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:40.577707 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:40.577675 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:40.681157 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:40.681121 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:42.338156 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:42.338116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:04:44.426317 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:44.426280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:51.890820 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:51.890779 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:51.989134 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:51.989102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:59.591451 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:59.591408 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:04:59.650088 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:04:59.650054 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:01.309134 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:01.309100 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:01.409865 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:01.409831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:08.741126 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.741091 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v"] Jul 06 16:05:08.748222 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.748188 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.751818 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.751789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3159bb15\"/\"default-dockercfg-btfdz\"" Jul 06 16:05:08.751962 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.751789 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3159bb15\"/\"openshift-service-ca.crt\"" Jul 06 16:05:08.751962 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.751819 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3159bb15\"/\"e2e-lifecycle-model-3159bb15-kserve-self-signed-certs\"" Jul 06 16:05:08.751962 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.751907 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3159bb15\"/\"kube-root-ca.crt\"" Jul 06 16:05:08.758043 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.758016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v"] Jul 06 16:05:08.783969 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.783936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:08.875099 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.875060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-dshm\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.875298 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.875107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tmp-dir\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.875298 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.875219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-model-cache\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.875298 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.875250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tls-certs\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.875298 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.875275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-home\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.875450 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.875301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9nw4\" (UniqueName: \"kubernetes.io/projected/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-kube-api-access-m9nw4\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976226 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-model-cache\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976226 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tls-certs\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-home\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976260 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9nw4\" (UniqueName: \"kubernetes.io/projected/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-kube-api-access-m9nw4\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-dshm\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tmp-dir\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976716 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976692 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-model-cache\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976777 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976722 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-home\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.976777 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.976754 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tmp-dir\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.978684 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.978658 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-dshm\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.978923 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.978907 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tls-certs\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:08.989686 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:08.989660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9nw4\" (UniqueName: \"kubernetes.io/projected/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-kube-api-access-m9nw4\") pod \"e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:09.058579 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:09.058538 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:09.202874 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:09.202850 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v"] Jul 06 16:05:09.205258 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:05:09.205232 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod095dbc42_d4ef_40c9_b4c5_9c3d76c735b3.slice/crio-ccb07ea106e86afaa1bbbe735513bea3876e898e2a2072959b168a07b4da13ac WatchSource:0}: Error finding container ccb07ea106e86afaa1bbbe735513bea3876e898e2a2072959b168a07b4da13ac: Status 404 returned error can't find the container with id ccb07ea106e86afaa1bbbe735513bea3876e898e2a2072959b168a07b4da13ac Jul 06 16:05:09.690358 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:09.690321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" event={"ID":"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3","Type":"ContainerStarted","Data":"0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821"} Jul 06 16:05:09.690358 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:09.690367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" event={"ID":"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3","Type":"ContainerStarted","Data":"ccb07ea106e86afaa1bbbe735513bea3876e898e2a2072959b168a07b4da13ac"} Jul 06 16:05:09.707470 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:09.707416 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" podStartSLOduration=1.707401807 podStartE2EDuration="1.707401807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:05:09.706117891 +0000 UTC m=+1878.640867966" watchObservedRunningTime="2026-07-06 16:05:09.707401807 +0000 UTC m=+1878.642151904" Jul 06 16:05:11.836914 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:11.836834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:05:19.059551 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:19.059506 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:19.059551 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:19.059552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:19.072148 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:19.072116 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:19.749855 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:19.749822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:05:24.933596 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:24.933557 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:05:27.187989 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:27.187949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:27.523060 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:27.522970 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:32.021361 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:32.021316 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:37.653381 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:37.653347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:05:43.261417 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:43.261378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:05:46.537899 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:46.537867 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:05:48.239797 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:48.239758 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v"] Jul 06 16:05:48.335567 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:05:48.335536 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:03.260016 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.259960 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" podUID="095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" containerName="main" containerID="cri-o://0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821" gracePeriod=45 Jul 06 16:06:03.510448 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.510398 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:06:03.573341 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573315 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-model-cache\") pod \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " Jul 06 16:06:03.573514 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573370 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tls-certs\") pod \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " Jul 06 16:06:03.573514 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573401 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tmp-dir\") pod \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " Jul 06 16:06:03.573637 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573517 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-dshm\") pod \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " Jul 06 16:06:03.573637 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573579 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-home\") pod \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " Jul 06 16:06:03.573637 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573616 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9nw4\" (UniqueName: \"kubernetes.io/projected/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-kube-api-access-m9nw4\") pod \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\" (UID: \"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3\") " Jul 06 16:06:03.573798 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573650 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" (UID: "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:06:03.573798 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573660 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" (UID: "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:06:03.573913 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573854 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-home" (OuterVolumeSpecName: "home") pod "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" (UID: "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:06:03.573913 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573873 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-model-cache\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:06:03.573913 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.573893 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tmp-dir\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:06:03.575831 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.575805 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-dshm" (OuterVolumeSpecName: "dshm") pod "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" (UID: "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:06:03.575931 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.575868 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-kube-api-access-m9nw4" (OuterVolumeSpecName: "kube-api-access-m9nw4") pod "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" (UID: "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3"). InnerVolumeSpecName "kube-api-access-m9nw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:06:03.575977 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.575919 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" (UID: "095dbc42-d4ef-40c9-b4c5-9c3d76c735b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:06:03.674817 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.674783 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-tls-certs\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:06:03.674817 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.674816 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-dshm\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:06:03.674992 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.674828 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-home\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:06:03.674992 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.674842 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9nw4\" (UniqueName: \"kubernetes.io/projected/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3-kube-api-access-m9nw4\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:06:03.943978 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.943952 2575 generic.go:356] "Generic (PLEG): container finished" podID="095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" containerID="0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821" exitCode=0 Jul 06 16:06:03.944117 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.943993 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" Jul 06 16:06:03.944117 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.944009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" event={"ID":"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3","Type":"ContainerDied","Data":"0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821"} Jul 06 16:06:03.944117 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.944042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v" event={"ID":"095dbc42-d4ef-40c9-b4c5-9c3d76c735b3","Type":"ContainerDied","Data":"ccb07ea106e86afaa1bbbe735513bea3876e898e2a2072959b168a07b4da13ac"} Jul 06 16:06:03.944117 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.944061 2575 scope.go:117] "RemoveContainer" containerID="0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821" Jul 06 16:06:03.952465 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.952436 2575 scope.go:117] "RemoveContainer" containerID="0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821" Jul 06 16:06:03.952754 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:06:03.952735 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821\": container with ID starting with 0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821 not found: ID does not exist" containerID="0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821" Jul 06 16:06:03.952810 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.952763 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821"} err="failed to get container status \"0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821\": rpc error: code = NotFound desc = could not find container \"0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821\": container with ID starting with 0da5c84fe26273c2b127a4a17e7a227d81d68050fee6cf6d1b4317c63c025821 not found: ID does not exist" Jul 06 16:06:03.960730 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.960686 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v"] Jul 06 16:06:03.962481 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:03.962463 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3159bb15/e2e-lifecycle-model-3159bb15-kserve-c5549d87f-x442v"] Jul 06 16:06:05.602935 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:05.602891 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" path="/var/lib/kubelet/pods/095dbc42-d4ef-40c9-b4c5-9c3d76c735b3/volumes" Jul 06 16:06:24.729138 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:24.729102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:26.310507 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:26.310468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:28.037909 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:28.037878 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:06:31.748202 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:31.748150 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:31.848764 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:31.848726 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:32.025231 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:32.025136 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:32.262150 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:32.262099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:33.324965 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:33.324932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:33.426264 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:33.426232 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:33.525750 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:33.525726 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:34.315405 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:34.315372 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:34.410291 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:34.410262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:34.513308 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:34.513278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:35.953855 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:35.953822 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:36.872792 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:36.872758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:06:44.526532 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:44.526450 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:44.590949 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:44.590919 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:49.912409 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:49.912349 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:49.974388 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:49.974362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:50.086462 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:50.086432 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:06:50.174544 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:50.174471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:06:50.177734 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:06:50.177711 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:00.711438 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:00.711409 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:01.092626 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:01.092586 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:22.104058 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:22.104019 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:28.875931 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:28.875890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:28.976575 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:28.976545 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:29.239325 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:29.239245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:31.044784 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:31.044741 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:31.149152 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:31.149117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:32.864113 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:32.864080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:32.946511 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:32.946471 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:33.045937 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:33.045900 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:33.440372 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:33.440342 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:33.579390 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:33.579310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:33.683987 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:33.683954 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:37.354068 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:37.354018 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:37.731532 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:37.731418 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:38.082695 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:38.082658 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:38.472557 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:38.472462 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:39.054713 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:39.054660 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:07:44.778790 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:44.778753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:44.878159 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:44.878103 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:45.093239 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.093198 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:45.147177 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.146911 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk"] Jul 06 16:07:45.147483 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.147464 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" containerName="main" Jul 06 16:07:45.147586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.147486 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" containerName="main" Jul 06 16:07:45.147649 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.147586 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="095dbc42-d4ef-40c9-b4c5-9c3d76c735b3" containerName="main" Jul 06 16:07:45.151071 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.151036 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.157229 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.157195 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hz5zw\"" Jul 06 16:07:45.157398 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.157237 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-785a99-serving-cert\"" Jul 06 16:07:45.164275 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.164240 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk"] Jul 06 16:07:45.305613 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.305577 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e8159-e11f-451e-aab1-8f6846c93ca8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.305806 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.305643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.305806 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.305757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqh8h\" (UniqueName: \"kubernetes.io/projected/b75e8159-e11f-451e-aab1-8f6846c93ca8-kube-api-access-cqh8h\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.406411 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.406308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.406411 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.406393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqh8h\" (UniqueName: \"kubernetes.io/projected/b75e8159-e11f-451e-aab1-8f6846c93ca8-kube-api-access-cqh8h\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.406649 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.406447 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e8159-e11f-451e-aab1-8f6846c93ca8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.406649 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:07:45.406460 2575 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-785a99-serving-cert: secret "maas-api-e2e-inf-b-785a99-serving-cert" not found Jul 06 16:07:45.406649 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:07:45.406521 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls podName:b75e8159-e11f-451e-aab1-8f6846c93ca8 nodeName:}" failed. No retries permitted until 2026-07-06 16:07:45.906503723 +0000 UTC m=+2034.841253749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls") pod "maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" (UID: "b75e8159-e11f-451e-aab1-8f6846c93ca8") : secret "maas-api-e2e-inf-b-785a99-serving-cert" not found Jul 06 16:07:45.407128 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.407105 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e8159-e11f-451e-aab1-8f6846c93ca8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.418503 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.418477 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqh8h\" (UniqueName: \"kubernetes.io/projected/b75e8159-e11f-451e-aab1-8f6846c93ca8-kube-api-access-cqh8h\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.596602 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.596567 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:45.803555 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.803525 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:45.891320 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.891277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:45.911079 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.911041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:45.913640 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:45.913621 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls\") pod \"maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:46.065844 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:46.065754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:46.205280 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:46.205251 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk"] Jul 06 16:07:46.207917 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:07:46.207888 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75e8159_e11f_451e_aab1_8f6846c93ca8.slice/crio-2fcb53daf608eda8dad8e27414f3728a9331ad8aaa6941933cebd7161e4a04fc WatchSource:0}: Error finding container 2fcb53daf608eda8dad8e27414f3728a9331ad8aaa6941933cebd7161e4a04fc: Status 404 returned error can't find the container with id 2fcb53daf608eda8dad8e27414f3728a9331ad8aaa6941933cebd7161e4a04fc Jul 06 16:07:46.209658 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:46.209630 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:07:46.400115 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:46.400013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" event={"ID":"b75e8159-e11f-451e-aab1-8f6846c93ca8","Type":"ContainerStarted","Data":"2fcb53daf608eda8dad8e27414f3728a9331ad8aaa6941933cebd7161e4a04fc"} Jul 06 16:07:48.053926 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:48.053885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:07:49.332074 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:49.332038 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:49.418965 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:49.418925 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" event={"ID":"b75e8159-e11f-451e-aab1-8f6846c93ca8","Type":"ContainerStarted","Data":"c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91"} Jul 06 16:07:49.419191 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:49.419051 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:49.476651 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:49.476585 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" podStartSLOduration=2.222119244 podStartE2EDuration="4.476562049s" totalImagesPullingTime="2.254442805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:07:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:07:49.473115629 +0000 UTC m=+2038.407865676" watchObservedRunningTime="2026-07-06 16:07:49.476562049 +0000 UTC m=+2038.411312097" Jul 06 16:07:55.429733 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:55.429704 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:07:55.766955 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:55.766872 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:07:55.862879 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:07:55.862847 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:01.908885 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.908843 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb"] Jul 06 16:08:01.912462 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.912439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:01.916824 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.916795 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-afeb72\"/\"openshift-service-ca.crt\"" Jul 06 16:08:01.918234 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.918215 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-afeb72\"/\"default-dockercfg-qdt97\"" Jul 06 16:08:01.918349 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.918255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-afeb72\"/\"test-model-afeb72-kserve-self-signed-certs\"" Jul 06 16:08:01.920987 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.920968 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-afeb72\"/\"kube-root-ca.crt\"" Jul 06 16:08:01.957336 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.957286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-home\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:01.957550 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.957382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gpgp\" (UniqueName: \"kubernetes.io/projected/a5f1c74a-26c4-4bab-a42b-3a525ea20853-kube-api-access-8gpgp\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:01.957550 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.957411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-model-cache\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:01.957550 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.957435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-dshm\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:01.957785 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.957551 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tls-certs\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:01.957785 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.957596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tmp-dir\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:01.961536 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.961506 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb"] Jul 06 16:08:01.981296 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:01.981245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:02.058303 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058265 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-model-cache\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058490 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-dshm\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058490 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tls-certs\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058490 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tmp-dir\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058490 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-home\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058706 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gpgp\" (UniqueName: \"kubernetes.io/projected/a5f1c74a-26c4-4bab-a42b-3a525ea20853-kube-api-access-8gpgp\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058761 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058712 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-model-cache\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058818 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058792 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tmp-dir\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.058880 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.058855 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-home\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.060857 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.060833 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-dshm\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.060994 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.060976 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tls-certs\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.092580 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.092546 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gpgp\" (UniqueName: \"kubernetes.io/projected/a5f1c74a-26c4-4bab-a42b-3a525ea20853-kube-api-access-8gpgp\") pod \"test-model-afeb72-kserve-7f946bc947-8rpnb\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.221454 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.221351 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:02.382909 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.382869 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb"] Jul 06 16:08:02.386244 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:08:02.386217 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f1c74a_26c4_4bab_a42b_3a525ea20853.slice/crio-813f717b9d1f5f0a0e04ca216e2286d3a4ed2c02158aa44a819c098f6884522b WatchSource:0}: Error finding container 813f717b9d1f5f0a0e04ca216e2286d3a4ed2c02158aa44a819c098f6884522b: Status 404 returned error can't find the container with id 813f717b9d1f5f0a0e04ca216e2286d3a4ed2c02158aa44a819c098f6884522b Jul 06 16:08:02.479836 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.479745 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" event={"ID":"a5f1c74a-26c4-4bab-a42b-3a525ea20853","Type":"ContainerStarted","Data":"e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db"} Jul 06 16:08:02.479836 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.479796 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" event={"ID":"a5f1c74a-26c4-4bab-a42b-3a525ea20853","Type":"ContainerStarted","Data":"813f717b9d1f5f0a0e04ca216e2286d3a4ed2c02158aa44a819c098f6884522b"} Jul 06 16:08:02.534608 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:02.534556 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" podStartSLOduration=1.534537946 podStartE2EDuration="1.534537946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:08:02.533472717 +0000 UTC m=+2051.468222764" watchObservedRunningTime="2026-07-06 16:08:02.534537946 +0000 UTC m=+2051.469287992" Jul 06 16:08:05.370193 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:05.370134 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:08:07.000921 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:07.000889 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:07.110535 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:07.110492 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:12.221937 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:12.221850 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:12.221937 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:12.221913 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:12.234237 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:12.234212 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:12.537826 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:12.537799 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:16.001063 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:16.001025 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:08:21.108616 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:21.108580 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:26.855434 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:26.855388 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:08:38.426572 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:38.426536 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb"] Jul 06 16:08:38.441404 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:38.441362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:39.351828 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:39.351795 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:40.540453 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:40.540425 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:40.625808 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:40.625772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:45.020856 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:45.020820 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:08:46.039303 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:46.039266 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:08:53.447530 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.447488 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" podUID="a5f1c74a-26c4-4bab-a42b-3a525ea20853" containerName="main" containerID="cri-o://e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db" gracePeriod=45 Jul 06 16:08:53.702749 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.702692 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:53.717495 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.717464 2575 generic.go:356] "Generic (PLEG): container finished" podID="a5f1c74a-26c4-4bab-a42b-3a525ea20853" containerID="e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db" exitCode=0 Jul 06 16:08:53.717653 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.717502 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" Jul 06 16:08:53.717653 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.717561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" event={"ID":"a5f1c74a-26c4-4bab-a42b-3a525ea20853","Type":"ContainerDied","Data":"e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db"} Jul 06 16:08:53.717653 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.717604 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb" event={"ID":"a5f1c74a-26c4-4bab-a42b-3a525ea20853","Type":"ContainerDied","Data":"813f717b9d1f5f0a0e04ca216e2286d3a4ed2c02158aa44a819c098f6884522b"} Jul 06 16:08:53.717653 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.717621 2575 scope.go:117] "RemoveContainer" containerID="e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db" Jul 06 16:08:53.726917 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.726900 2575 scope.go:117] "RemoveContainer" containerID="e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db" Jul 06 16:08:53.727220 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:08:53.727197 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db\": container with ID starting with e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db not found: ID does not exist" containerID="e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db" Jul 06 16:08:53.727323 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.727225 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db"} err="failed to get container status \"e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db\": rpc error: code = NotFound desc = could not find container \"e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db\": container with ID starting with e2c1928bb280c5c644b3230efc23a075edb54f049f8ea1536e15cbaba8dff5db not found: ID does not exist" Jul 06 16:08:53.830066 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830035 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-home\") pod \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " Jul 06 16:08:53.830283 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830080 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tls-certs\") pod \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " Jul 06 16:08:53.830283 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tmp-dir\") pod \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " Jul 06 16:08:53.830283 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830153 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gpgp\" (UniqueName: \"kubernetes.io/projected/a5f1c74a-26c4-4bab-a42b-3a525ea20853-kube-api-access-8gpgp\") pod \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " Jul 06 16:08:53.830283 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-dshm\") pod \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " Jul 06 16:08:53.830283 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830233 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-model-cache\") pod \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\" (UID: \"a5f1c74a-26c4-4bab-a42b-3a525ea20853\") " Jul 06 16:08:53.830557 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830384 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-home" (OuterVolumeSpecName: "home") pod "a5f1c74a-26c4-4bab-a42b-3a525ea20853" (UID: "a5f1c74a-26c4-4bab-a42b-3a525ea20853"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:08:53.830557 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830420 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5f1c74a-26c4-4bab-a42b-3a525ea20853" (UID: "a5f1c74a-26c4-4bab-a42b-3a525ea20853"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:08:53.830557 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830523 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tmp-dir\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:08:53.830557 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830542 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-home\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:08:53.830703 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.830562 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5f1c74a-26c4-4bab-a42b-3a525ea20853" (UID: "a5f1c74a-26c4-4bab-a42b-3a525ea20853"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:08:53.832489 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.832466 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-dshm" (OuterVolumeSpecName: "dshm") pod "a5f1c74a-26c4-4bab-a42b-3a525ea20853" (UID: "a5f1c74a-26c4-4bab-a42b-3a525ea20853"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:08:53.832571 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.832528 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5f1c74a-26c4-4bab-a42b-3a525ea20853" (UID: "a5f1c74a-26c4-4bab-a42b-3a525ea20853"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:08:53.832571 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.832540 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f1c74a-26c4-4bab-a42b-3a525ea20853-kube-api-access-8gpgp" (OuterVolumeSpecName: "kube-api-access-8gpgp") pod "a5f1c74a-26c4-4bab-a42b-3a525ea20853" (UID: "a5f1c74a-26c4-4bab-a42b-3a525ea20853"). InnerVolumeSpecName "kube-api-access-8gpgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:08:53.931815 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.931776 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-model-cache\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:08:53.931815 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.931810 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f1c74a-26c4-4bab-a42b-3a525ea20853-tls-certs\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:08:53.931815 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.931819 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gpgp\" (UniqueName: \"kubernetes.io/projected/a5f1c74a-26c4-4bab-a42b-3a525ea20853-kube-api-access-8gpgp\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:08:53.932042 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:53.931828 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f1c74a-26c4-4bab-a42b-3a525ea20853-dshm\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:08:54.039091 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:54.039057 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb"] Jul 06 16:08:54.043411 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:54.043388 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-afeb72/test-model-afeb72-kserve-7f946bc947-8rpnb"] Jul 06 16:08:55.602727 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:08:55.602693 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5f1c74a-26c4-4bab-a42b-3a525ea20853" path="/var/lib/kubelet/pods/a5f1c74a-26c4-4bab-a42b-3a525ea20853/volumes" Jul 06 16:09:22.577643 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:22.577599 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:22.856125 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:22.856016 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:23.217076 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:23.216982 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:23.410504 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:23.410467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:23.807674 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:23.807631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:23.904694 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:23.904648 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:24.013197 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.013133 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:24.108830 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.108739 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk"] Jul 06 16:09:24.109051 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.109006 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" podUID="b75e8159-e11f-451e-aab1-8f6846c93ca8" containerName="maas-api" containerID="cri-o://c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91" gracePeriod=30 Jul 06 16:09:24.153777 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.153746 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:24.359769 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.359711 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:09:24.392720 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.392690 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e8159-e11f-451e-aab1-8f6846c93ca8-openshift-service-ca\") pod \"b75e8159-e11f-451e-aab1-8f6846c93ca8\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " Jul 06 16:09:24.392877 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.392768 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqh8h\" (UniqueName: \"kubernetes.io/projected/b75e8159-e11f-451e-aab1-8f6846c93ca8-kube-api-access-cqh8h\") pod \"b75e8159-e11f-451e-aab1-8f6846c93ca8\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " Jul 06 16:09:24.392877 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.392840 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls\") pod \"b75e8159-e11f-451e-aab1-8f6846c93ca8\" (UID: \"b75e8159-e11f-451e-aab1-8f6846c93ca8\") " Jul 06 16:09:24.393108 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.393086 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75e8159-e11f-451e-aab1-8f6846c93ca8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b75e8159-e11f-451e-aab1-8f6846c93ca8" (UID: "b75e8159-e11f-451e-aab1-8f6846c93ca8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:09:24.395129 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.395105 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75e8159-e11f-451e-aab1-8f6846c93ca8-kube-api-access-cqh8h" (OuterVolumeSpecName: "kube-api-access-cqh8h") pod "b75e8159-e11f-451e-aab1-8f6846c93ca8" (UID: "b75e8159-e11f-451e-aab1-8f6846c93ca8"). InnerVolumeSpecName "kube-api-access-cqh8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:09:24.395244 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.395142 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b75e8159-e11f-451e-aab1-8f6846c93ca8" (UID: "b75e8159-e11f-451e-aab1-8f6846c93ca8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:09:24.493492 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.493453 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75e8159-e11f-451e-aab1-8f6846c93ca8-maas-api-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:09:24.493492 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.493485 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75e8159-e11f-451e-aab1-8f6846c93ca8-openshift-service-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:09:24.493492 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.493495 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqh8h\" (UniqueName: \"kubernetes.io/projected/b75e8159-e11f-451e-aab1-8f6846c93ca8-kube-api-access-cqh8h\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:09:24.869750 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.869714 2575 generic.go:356] "Generic (PLEG): container finished" podID="b75e8159-e11f-451e-aab1-8f6846c93ca8" containerID="c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91" exitCode=0 Jul 06 16:09:24.869750 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.869754 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" Jul 06 16:09:24.870247 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.869758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" event={"ID":"b75e8159-e11f-451e-aab1-8f6846c93ca8","Type":"ContainerDied","Data":"c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91"} Jul 06 16:09:24.870247 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.869892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk" event={"ID":"b75e8159-e11f-451e-aab1-8f6846c93ca8","Type":"ContainerDied","Data":"2fcb53daf608eda8dad8e27414f3728a9331ad8aaa6941933cebd7161e4a04fc"} Jul 06 16:09:24.870247 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.869923 2575 scope.go:117] "RemoveContainer" containerID="c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91" Jul 06 16:09:24.879344 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.879317 2575 scope.go:117] "RemoveContainer" containerID="c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91" Jul 06 16:09:24.879724 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:09:24.879694 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91\": container with ID starting with c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91 not found: ID does not exist" containerID="c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91" Jul 06 16:09:24.879800 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.879734 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91"} err="failed to get container status \"c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91\": rpc error: code = NotFound desc = could not find container \"c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91\": container with ID starting with c531b21790bd5e7a069579b51c53f9b7e5a235dc2f6582542f2040f423a28f91 not found: ID does not exist" Jul 06 16:09:24.892969 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.892940 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk"] Jul 06 16:09:24.898854 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:24.898828 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-785a99-6b5fcf4b76-4v6hk"] Jul 06 16:09:25.603707 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:25.603672 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b75e8159-e11f-451e-aab1-8f6846c93ca8" path="/var/lib/kubelet/pods/b75e8159-e11f-451e-aab1-8f6846c93ca8/volumes" Jul 06 16:09:26.640616 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:26.640576 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:09:29.741142 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:29.741100 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:29.870358 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:29.870323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:29.972080 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:29.972048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:30.168893 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:30.168855 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:30.436479 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:30.436395 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:30.798203 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:30.798156 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:31.127104 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:31.127021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:31.489413 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:31.489323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:34.450843 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.450805 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:34.492861 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.492825 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw"] Jul 06 16:09:34.493431 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.493411 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5f1c74a-26c4-4bab-a42b-3a525ea20853" containerName="main" Jul 06 16:09:34.493482 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.493435 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5f1c74a-26c4-4bab-a42b-3a525ea20853" containerName="main" Jul 06 16:09:34.493482 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.493448 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b75e8159-e11f-451e-aab1-8f6846c93ca8" containerName="maas-api" Jul 06 16:09:34.493482 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.493456 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b75e8159-e11f-451e-aab1-8f6846c93ca8" containerName="maas-api" Jul 06 16:09:34.493579 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.493568 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5f1c74a-26c4-4bab-a42b-3a525ea20853" containerName="main" Jul 06 16:09:34.493613 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.493582 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b75e8159-e11f-451e-aab1-8f6846c93ca8" containerName="maas-api" Jul 06 16:09:34.498394 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.498374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.500821 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.500798 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-34ada1-serving-cert\"" Jul 06 16:09:34.500926 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.500809 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hz5zw\"" Jul 06 16:09:34.511057 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.511033 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw"] Jul 06 16:09:34.550079 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.550005 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:34.585767 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.585736 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-maas-api-tls\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.585947 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.585788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.585947 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.585839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6xw2\" (UniqueName: \"kubernetes.io/projected/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-kube-api-access-t6xw2\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.686507 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.686472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-maas-api-tls\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.686684 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.686521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.686684 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.686550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6xw2\" (UniqueName: \"kubernetes.io/projected/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-kube-api-access-t6xw2\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.687241 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.687216 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-openshift-service-ca\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.689132 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.689114 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-maas-api-tls\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.695356 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.695330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6xw2\" (UniqueName: \"kubernetes.io/projected/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-kube-api-access-t6xw2\") pod \"maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.750451 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.750357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:34.808663 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.808634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:34.944764 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:34.944738 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw"] Jul 06 16:09:34.947598 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:09:34.947570 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ada6d3_fb9a_4228_b2bd_ae45fc5283e2.slice/crio-a3d1bf72645c3ede12bb35788c142497d83ecf3304015fd1c2bb186fa2d2a76b WatchSource:0}: Error finding container a3d1bf72645c3ede12bb35788c142497d83ecf3304015fd1c2bb186fa2d2a76b: Status 404 returned error can't find the container with id a3d1bf72645c3ede12bb35788c142497d83ecf3304015fd1c2bb186fa2d2a76b Jul 06 16:09:35.092600 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:35.092567 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:35.194022 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:35.193988 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:35.930267 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:35.930233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" event={"ID":"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2","Type":"ContainerStarted","Data":"604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1"} Jul 06 16:09:35.930648 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:35.930283 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" event={"ID":"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2","Type":"ContainerStarted","Data":"a3d1bf72645c3ede12bb35788c142497d83ecf3304015fd1c2bb186fa2d2a76b"} Jul 06 16:09:35.930648 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:35.930331 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:35.949141 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:35.949093 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" podStartSLOduration=1.451579091 podStartE2EDuration="1.949075857s" totalImagesPullingTime="497.496766ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:09:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:09:35.94801496 +0000 UTC m=+2144.882765007" watchObservedRunningTime="2026-07-06 16:09:35.949075857 +0000 UTC m=+2144.883825904" Jul 06 16:09:38.948702 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:38.948667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:39.419377 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:39.419339 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:39.544322 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:39.544283 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:09:39.740805 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:39.740717 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:39.864297 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:39.864262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:41.940785 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:41.940712 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:42.442822 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.442794 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:42.511288 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.511252 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn"] Jul 06 16:09:42.514772 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.514756 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.518698 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.518673 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-05b5d9-serving-cert\"" Jul 06 16:09:42.545047 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.545018 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn"] Jul 06 16:09:42.548360 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.548333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:42.660144 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.660098 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:42.663270 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.663245 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dzk\" (UniqueName: \"kubernetes.io/projected/1a4793a2-ab31-43f2-b1ee-158140a206f1-kube-api-access-s6dzk\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.663406 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.663303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.663406 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.663392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4793a2-ab31-43f2-b1ee-158140a206f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.764991 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.764890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4793a2-ab31-43f2-b1ee-158140a206f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.764991 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.764977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dzk\" (UniqueName: \"kubernetes.io/projected/1a4793a2-ab31-43f2-b1ee-158140a206f1-kube-api-access-s6dzk\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.765230 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.765039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.765230 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:09:42.765206 2575 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-05b5d9-serving-cert: secret "maas-api-e2e-api-b-05b5d9-serving-cert" not found Jul 06 16:09:42.765313 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:09:42.765299 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls podName:1a4793a2-ab31-43f2-b1ee-158140a206f1 nodeName:}" failed. No retries permitted until 2026-07-06 16:09:43.265278018 +0000 UTC m=+2152.200028059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls") pod "maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" (UID: "1a4793a2-ab31-43f2-b1ee-158140a206f1") : secret "maas-api-e2e-api-b-05b5d9-serving-cert" not found Jul 06 16:09:42.765673 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.765655 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4793a2-ab31-43f2-b1ee-158140a206f1-openshift-service-ca\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:42.780575 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:42.780546 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dzk\" (UniqueName: \"kubernetes.io/projected/1a4793a2-ab31-43f2-b1ee-158140a206f1-kube-api-access-s6dzk\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:43.272438 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:43.272399 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:43.272906 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:09:43.272563 2575 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-b-05b5d9-serving-cert: secret "maas-api-e2e-api-b-05b5d9-serving-cert" not found Jul 06 16:09:43.272906 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:09:43.272650 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls podName:1a4793a2-ab31-43f2-b1ee-158140a206f1 nodeName:}" failed. No retries permitted until 2026-07-06 16:09:44.272630065 +0000 UTC m=+2153.207380091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls") pod "maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" (UID: "1a4793a2-ab31-43f2-b1ee-158140a206f1") : secret "maas-api-e2e-api-b-05b5d9-serving-cert" not found Jul 06 16:09:43.520886 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:43.520821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:43.661103 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:43.661070 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:43.814130 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:43.814089 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:44.281580 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:44.281539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:44.284219 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:44.284190 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls\") pod \"maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:44.329007 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:44.328963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:44.466023 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:44.465988 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn"] Jul 06 16:09:44.468952 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:09:44.468927 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4793a2_ab31_43f2_b1ee_158140a206f1.slice/crio-2d84ed876bc9d43e19b9091da9171d33f3e84909d243f6a291157d278a6af7ec WatchSource:0}: Error finding container 2d84ed876bc9d43e19b9091da9171d33f3e84909d243f6a291157d278a6af7ec: Status 404 returned error can't find the container with id 2d84ed876bc9d43e19b9091da9171d33f3e84909d243f6a291157d278a6af7ec Jul 06 16:09:44.984431 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:44.984392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" event={"ID":"1a4793a2-ab31-43f2-b1ee-158140a206f1","Type":"ContainerStarted","Data":"2d84ed876bc9d43e19b9091da9171d33f3e84909d243f6a291157d278a6af7ec"} Jul 06 16:09:45.991489 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:45.991395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" event={"ID":"1a4793a2-ab31-43f2-b1ee-158140a206f1","Type":"ContainerStarted","Data":"94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642"} Jul 06 16:09:45.991936 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:45.991525 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:46.012638 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:46.012586 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" podStartSLOduration=2.904684579 podStartE2EDuration="4.012570883s" totalImagesPullingTime="1.107886304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:09:46.010836087 +0000 UTC m=+2154.945586154" watchObservedRunningTime="2026-07-06 16:09:46.012570883 +0000 UTC m=+2154.947320942" Jul 06 16:09:46.995311 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:46.995273 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:48.145670 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:48.145625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:09:52.002056 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:52.001978 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:09:52.689945 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:52.689911 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:52.793887 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:52.793846 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:55.382064 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:55.382029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:55.708840 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:55.708744 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:55.807924 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:55.807884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:55.914408 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:55.914376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:56.048707 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.048674 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw"] Jul 06 16:09:56.048931 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.048904 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" podUID="48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" containerName="maas-api" containerID="cri-o://604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1" gracePeriod=30 Jul 06 16:09:56.086681 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.086646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:09:56.295081 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.295055 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:56.392315 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.392227 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-openshift-service-ca\") pod \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " Jul 06 16:09:56.392315 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.392274 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-maas-api-tls\") pod \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " Jul 06 16:09:56.392801 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.392336 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6xw2\" (UniqueName: \"kubernetes.io/projected/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-kube-api-access-t6xw2\") pod \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\" (UID: \"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2\") " Jul 06 16:09:56.392801 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.392686 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" (UID: "48ada6d3-fb9a-4228-b2bd-ae45fc5283e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:09:56.394600 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.394576 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-kube-api-access-t6xw2" (OuterVolumeSpecName: "kube-api-access-t6xw2") pod "48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" (UID: "48ada6d3-fb9a-4228-b2bd-ae45fc5283e2"). InnerVolumeSpecName "kube-api-access-t6xw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:09:56.394690 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.394638 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" (UID: "48ada6d3-fb9a-4228-b2bd-ae45fc5283e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:09:56.493482 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.493449 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6xw2\" (UniqueName: \"kubernetes.io/projected/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-kube-api-access-t6xw2\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:09:56.493482 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.493478 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-openshift-service-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:09:56.493482 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:56.493487 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2-maas-api-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:09:57.043185 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.043127 2575 generic.go:356] "Generic (PLEG): container finished" podID="48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" containerID="604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1" exitCode=0 Jul 06 16:09:57.043362 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.043189 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" Jul 06 16:09:57.043362 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.043205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" event={"ID":"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2","Type":"ContainerDied","Data":"604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1"} Jul 06 16:09:57.043362 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.043263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw" event={"ID":"48ada6d3-fb9a-4228-b2bd-ae45fc5283e2","Type":"ContainerDied","Data":"a3d1bf72645c3ede12bb35788c142497d83ecf3304015fd1c2bb186fa2d2a76b"} Jul 06 16:09:57.043362 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.043285 2575 scope.go:117] "RemoveContainer" containerID="604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1" Jul 06 16:09:57.052081 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.051912 2575 scope.go:117] "RemoveContainer" containerID="604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1" Jul 06 16:09:57.052244 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:09:57.052224 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1\": container with ID starting with 604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1 not found: ID does not exist" containerID="604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1" Jul 06 16:09:57.052296 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.052252 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1"} err="failed to get container status \"604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1\": rpc error: code = NotFound desc = could not find container \"604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1\": container with ID starting with 604f57a4c07f2174b2ba06d7937c7af52ef39ef324303266f693eb74d15e62a1 not found: ID does not exist" Jul 06 16:09:57.065716 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.065688 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw"] Jul 06 16:09:57.070289 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.070267 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-34ada1-7b9bb9bb95-ndngw"] Jul 06 16:09:57.603482 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:57.603447 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" path="/var/lib/kubelet/pods/48ada6d3-fb9a-4228-b2bd-ae45fc5283e2/volumes" Jul 06 16:09:59.243420 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:09:59.243380 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:10:03.020046 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:03.020011 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:03.101895 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:03.101858 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:03.578528 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:03.578489 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:03.685109 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:03.685081 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:03.832264 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:03.832131 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:04.273305 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.273210 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:04.465370 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.465310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:04.575222 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.575189 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn"] Jul 06 16:10:04.575444 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.575417 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" podUID="1a4793a2-ab31-43f2-b1ee-158140a206f1" containerName="maas-api" containerID="cri-o://94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642" gracePeriod=30 Jul 06 16:10:04.609091 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.609062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:04.822666 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.822640 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:10:04.964497 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.964409 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls\") pod \"1a4793a2-ab31-43f2-b1ee-158140a206f1\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " Jul 06 16:10:04.964497 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.964479 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4793a2-ab31-43f2-b1ee-158140a206f1-openshift-service-ca\") pod \"1a4793a2-ab31-43f2-b1ee-158140a206f1\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " Jul 06 16:10:04.964732 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.964507 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6dzk\" (UniqueName: \"kubernetes.io/projected/1a4793a2-ab31-43f2-b1ee-158140a206f1-kube-api-access-s6dzk\") pod \"1a4793a2-ab31-43f2-b1ee-158140a206f1\" (UID: \"1a4793a2-ab31-43f2-b1ee-158140a206f1\") " Jul 06 16:10:04.964899 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.964872 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4793a2-ab31-43f2-b1ee-158140a206f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a4793a2-ab31-43f2-b1ee-158140a206f1" (UID: "1a4793a2-ab31-43f2-b1ee-158140a206f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:10:04.966711 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.966687 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a4793a2-ab31-43f2-b1ee-158140a206f1" (UID: "1a4793a2-ab31-43f2-b1ee-158140a206f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:10:04.966820 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:04.966731 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4793a2-ab31-43f2-b1ee-158140a206f1-kube-api-access-s6dzk" (OuterVolumeSpecName: "kube-api-access-s6dzk") pod "1a4793a2-ab31-43f2-b1ee-158140a206f1" (UID: "1a4793a2-ab31-43f2-b1ee-158140a206f1"). InnerVolumeSpecName "kube-api-access-s6dzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:10:05.065606 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.065578 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a4793a2-ab31-43f2-b1ee-158140a206f1-openshift-service-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:10:05.065606 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.065604 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6dzk\" (UniqueName: \"kubernetes.io/projected/1a4793a2-ab31-43f2-b1ee-158140a206f1-kube-api-access-s6dzk\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:10:05.065606 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.065614 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a4793a2-ab31-43f2-b1ee-158140a206f1-maas-api-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:10:05.082082 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.082048 2575 generic.go:356] "Generic (PLEG): container finished" podID="1a4793a2-ab31-43f2-b1ee-158140a206f1" containerID="94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642" exitCode=0 Jul 06 16:10:05.082230 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.082105 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" Jul 06 16:10:05.082230 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.082136 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" event={"ID":"1a4793a2-ab31-43f2-b1ee-158140a206f1","Type":"ContainerDied","Data":"94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642"} Jul 06 16:10:05.082230 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.082210 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn" event={"ID":"1a4793a2-ab31-43f2-b1ee-158140a206f1","Type":"ContainerDied","Data":"2d84ed876bc9d43e19b9091da9171d33f3e84909d243f6a291157d278a6af7ec"} Jul 06 16:10:05.082349 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.082232 2575 scope.go:117] "RemoveContainer" containerID="94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642" Jul 06 16:10:05.090759 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.090741 2575 scope.go:117] "RemoveContainer" containerID="94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642" Jul 06 16:10:05.091022 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:10:05.091002 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642\": container with ID starting with 94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642 not found: ID does not exist" containerID="94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642" Jul 06 16:10:05.091091 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.091028 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642"} err="failed to get container status \"94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642\": rpc error: code = NotFound desc = could not find container \"94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642\": container with ID starting with 94aaec090b578a536e8d59df2c791b97ca54c479b206e4949c2a54f8cc06d642 not found: ID does not exist" Jul 06 16:10:05.112317 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.112283 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn"] Jul 06 16:10:05.120650 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.120620 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-05b5d9-7df755d565-ks6pn"] Jul 06 16:10:05.602843 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:05.602810 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4793a2-ab31-43f2-b1ee-158140a206f1" path="/var/lib/kubelet/pods/1a4793a2-ab31-43f2-b1ee-158140a206f1/volumes" Jul 06 16:10:11.146876 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:11.146841 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:11.540458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:11.540419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:11.789270 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:11.789232 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:11.921111 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:11.921029 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:12.068611 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:12.068578 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:12.492657 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:12.492623 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:12.916291 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:12.916231 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:13.986234 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:13.986201 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:10:15.831428 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.831397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:15.965357 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.965321 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r"] Jul 06 16:10:15.965712 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.965699 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" containerName="maas-api" Jul 06 16:10:15.965762 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.965714 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" containerName="maas-api" Jul 06 16:10:15.965762 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.965730 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4793a2-ab31-43f2-b1ee-158140a206f1" containerName="maas-api" Jul 06 16:10:15.965762 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.965736 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4793a2-ab31-43f2-b1ee-158140a206f1" containerName="maas-api" Jul 06 16:10:15.965853 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.965792 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4793a2-ab31-43f2-b1ee-158140a206f1" containerName="maas-api" Jul 06 16:10:15.965853 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.965804 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="48ada6d3-fb9a-4228-b2bd-ae45fc5283e2" containerName="maas-api" Jul 06 16:10:15.970046 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.970025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:15.981236 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.981209 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-c8a9d1-serving-cert\"" Jul 06 16:10:15.985462 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:15.985440 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-hz5zw\"" Jul 06 16:10:16.011184 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.011126 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r"] Jul 06 16:10:16.011965 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.011938 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:16.062892 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.062845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgcmg\" (UniqueName: \"kubernetes.io/projected/05749eb4-e78f-422f-aceb-5edb4d1690bf-kube-api-access-bgcmg\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.063093 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.062917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.063093 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.062966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05749eb4-e78f-422f-aceb-5edb4d1690bf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.163887 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.163791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcmg\" (UniqueName: \"kubernetes.io/projected/05749eb4-e78f-422f-aceb-5edb4d1690bf-kube-api-access-bgcmg\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.163887 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.163846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.163887 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.163884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05749eb4-e78f-422f-aceb-5edb4d1690bf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.164130 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:10:16.164027 2575 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-c8a9d1-serving-cert: secret "maas-api-e2e-shared-a-c8a9d1-serving-cert" not found Jul 06 16:10:16.164130 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:10:16.164125 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls podName:05749eb4-e78f-422f-aceb-5edb4d1690bf nodeName:}" failed. No retries permitted until 2026-07-06 16:10:16.664101806 +0000 UTC m=+2185.598851833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls") pod "maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" (UID: "05749eb4-e78f-422f-aceb-5edb4d1690bf") : secret "maas-api-e2e-shared-a-c8a9d1-serving-cert" not found Jul 06 16:10:16.164666 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.164647 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05749eb4-e78f-422f-aceb-5edb4d1690bf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.173285 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.173258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgcmg\" (UniqueName: \"kubernetes.io/projected/05749eb4-e78f-422f-aceb-5edb4d1690bf-kube-api-access-bgcmg\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.175045 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.175019 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:16.342736 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.342694 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:16.666885 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.666847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.669365 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.669344 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls\") pod \"maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:16.757406 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.757370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:16.880970 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:16.880932 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:17.028951 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:17.028924 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r"] Jul 06 16:10:17.031121 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:10:17.031093 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05749eb4_e78f_422f_aceb_5edb4d1690bf.slice/crio-4ac24e62757edb992c9aa349537a86f05277c8ea4888b925db8d86aad6e16ba0 WatchSource:0}: Error finding container 4ac24e62757edb992c9aa349537a86f05277c8ea4888b925db8d86aad6e16ba0: Status 404 returned error can't find the container with id 4ac24e62757edb992c9aa349537a86f05277c8ea4888b925db8d86aad6e16ba0 Jul 06 16:10:17.145051 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:17.145021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" event={"ID":"05749eb4-e78f-422f-aceb-5edb4d1690bf","Type":"ContainerStarted","Data":"4ac24e62757edb992c9aa349537a86f05277c8ea4888b925db8d86aad6e16ba0"} Jul 06 16:10:17.507877 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:17.507842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:18.150778 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:18.150743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" event={"ID":"05749eb4-e78f-422f-aceb-5edb4d1690bf","Type":"ContainerStarted","Data":"ce1fb49285ecc03f94d668076255c981061cb188b8ee28285be8cdb22722cc43"} Jul 06 16:10:18.151251 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:18.150855 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:18.171224 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:18.171142 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" podStartSLOduration=2.570621807 podStartE2EDuration="3.171124228s" totalImagesPullingTime="600.502421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:10:18.170566091 +0000 UTC m=+2187.105316213" watchObservedRunningTime="2026-07-06 16:10:18.171124228 +0000 UTC m=+2187.105874272" Jul 06 16:10:20.875115 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:20.875075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:21.166756 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:21.166660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:21.588112 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:21.588078 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:23.798211 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.798155 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:23.829948 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.829911 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9"] Jul 06 16:10:23.833195 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.833154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:23.837078 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.837043 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-b747bf-serving-cert\"" Jul 06 16:10:23.848541 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.848508 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9"] Jul 06 16:10:23.933504 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.932926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6shjv\" (UniqueName: \"kubernetes.io/projected/9a408186-f3c2-4925-8aa8-38be943e3596-kube-api-access-6shjv\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:23.933504 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.933024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a408186-f3c2-4925-8aa8-38be943e3596-maas-api-tls\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:23.933504 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:23.933135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a408186-f3c2-4925-8aa8-38be943e3596-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.033720 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.033677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a408186-f3c2-4925-8aa8-38be943e3596-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.033918 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.033734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6shjv\" (UniqueName: \"kubernetes.io/projected/9a408186-f3c2-4925-8aa8-38be943e3596-kube-api-access-6shjv\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.033918 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.033788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a408186-f3c2-4925-8aa8-38be943e3596-maas-api-tls\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.034546 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.034520 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a408186-f3c2-4925-8aa8-38be943e3596-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.036495 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.036462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a408186-f3c2-4925-8aa8-38be943e3596-maas-api-tls\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.042243 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.042216 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6shjv\" (UniqueName: \"kubernetes.io/projected/9a408186-f3c2-4925-8aa8-38be943e3596-kube-api-access-6shjv\") pod \"maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.122202 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.122083 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:24.144371 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.144332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:24.164279 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.164251 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:10:24.230865 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.230823 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:24.316621 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.316595 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9"] Jul 06 16:10:24.318680 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:10:24.318659 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a408186_f3c2_4925_8aa8_38be943e3596.slice/crio-e9e6f18b24b64e217735c729718e50a2d0c8274750523431b4ea62e37d1fc6a7 WatchSource:0}: Error finding container e9e6f18b24b64e217735c729718e50a2d0c8274750523431b4ea62e37d1fc6a7: Status 404 returned error can't find the container with id e9e6f18b24b64e217735c729718e50a2d0c8274750523431b4ea62e37d1fc6a7 Jul 06 16:10:24.569674 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.569640 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:24.761737 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:24.761705 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:25.186831 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:25.186795 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" event={"ID":"9a408186-f3c2-4925-8aa8-38be943e3596","Type":"ContainerStarted","Data":"4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16"} Jul 06 16:10:25.187226 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:25.186880 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" event={"ID":"9a408186-f3c2-4925-8aa8-38be943e3596","Type":"ContainerStarted","Data":"e9e6f18b24b64e217735c729718e50a2d0c8274750523431b4ea62e37d1fc6a7"} Jul 06 16:10:25.187226 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:25.186983 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:25.214286 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:25.214233 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" podStartSLOduration=1.8875545599999999 podStartE2EDuration="2.214217164s" totalImagesPullingTime="326.662604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:10:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:10:25.211007465 +0000 UTC m=+2194.145757512" watchObservedRunningTime="2026-07-06 16:10:25.214217164 +0000 UTC m=+2194.148967211" Jul 06 16:10:26.457048 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:26.457011 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:10:28.291316 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:28.291273 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:31.197503 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:31.197451 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:10:36.079298 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:36.079257 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:36.816110 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:36.816076 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:36.917812 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:36.917762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:37.715274 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:37.715245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:37.966304 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:37.966220 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:41.344592 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:41.344548 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:10:46.291318 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:46.291282 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:10:46.393463 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:10:46.393424 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:11:28.247488 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.247410 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks"] Jul 06 16:11:28.250814 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.250797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.253613 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.253592 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"kube-root-ca.crt\"" Jul 06 16:11:28.254458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.254424 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"openshift-service-ca.crt\"" Jul 06 16:11:28.254458 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.254441 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"auth-test-model-b747bf-kserve-self-signed-certs\"" Jul 06 16:11:28.254638 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.254552 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"default-dockercfg-qw6cg\"" Jul 06 16:11:28.261534 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.261508 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks"] Jul 06 16:11:28.288639 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.288612 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:11:28.420547 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.420510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29be4850-f11f-42c8-8b2e-5baeb09e4278-tls-certs\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.420547 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.420549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-home\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.420754 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.420599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmwb\" (UniqueName: \"kubernetes.io/projected/29be4850-f11f-42c8-8b2e-5baeb09e4278-kube-api-access-ljmwb\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.420754 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.420624 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-model-cache\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.420754 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.420646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-dshm\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.420754 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.420671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-tmp-dir\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521353 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-model-cache\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521353 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-dshm\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521353 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-tmp-dir\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521647 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521415 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29be4850-f11f-42c8-8b2e-5baeb09e4278-tls-certs\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521647 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-home\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521647 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmwb\" (UniqueName: \"kubernetes.io/projected/29be4850-f11f-42c8-8b2e-5baeb09e4278-kube-api-access-ljmwb\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521895 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521763 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-model-cache\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521895 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521826 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-tmp-dir\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.521895 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.521877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-home\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.523861 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.523839 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-dshm\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.524218 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.524201 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29be4850-f11f-42c8-8b2e-5baeb09e4278-tls-certs\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.530883 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.530856 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmwb\" (UniqueName: \"kubernetes.io/projected/29be4850-f11f-42c8-8b2e-5baeb09e4278-kube-api-access-ljmwb\") pod \"auth-test-model-b747bf-kserve-84c96cfb67-pkjks\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.560796 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.560759 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:28.698139 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:28.698111 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks"] Jul 06 16:11:28.700968 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:11:28.700942 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29be4850_f11f_42c8_8b2e_5baeb09e4278.slice/crio-6b2847808007e933105c2b8a8906053cdb712fae3a66f53ae37ca47178c42558 WatchSource:0}: Error finding container 6b2847808007e933105c2b8a8906053cdb712fae3a66f53ae37ca47178c42558: Status 404 returned error can't find the container with id 6b2847808007e933105c2b8a8906053cdb712fae3a66f53ae37ca47178c42558 Jul 06 16:11:29.479882 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:29.479846 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" event={"ID":"29be4850-f11f-42c8-8b2e-5baeb09e4278","Type":"ContainerStarted","Data":"63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261"} Jul 06 16:11:29.479882 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:29.479887 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" event={"ID":"29be4850-f11f-42c8-8b2e-5baeb09e4278","Type":"ContainerStarted","Data":"6b2847808007e933105c2b8a8906053cdb712fae3a66f53ae37ca47178c42558"} Jul 06 16:11:29.502902 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:29.502851 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" podStartSLOduration=1.502834037 podStartE2EDuration="1.502834037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:11:29.502364313 +0000 UTC m=+2258.437114360" watchObservedRunningTime="2026-07-06 16:11:29.502834037 +0000 UTC m=+2258.437584083" Jul 06 16:11:31.884938 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:31.884898 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:11:38.561587 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:38.561547 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:38.561587 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:38.561596 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:38.574210 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:38.574153 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:39.537753 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:39.537725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:11:41.493758 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:41.493720 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:11:41.562145 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:41.562113 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:11:54.984432 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:11:54.984390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:12:07.643839 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:12:07.643806 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:12:23.278391 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:12:23.278349 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:12:30.773098 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:12:30.773060 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:12:39.845082 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:12:39.845052 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:12:51.033767 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:12:51.033690 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:12:56.851767 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:12:56.851732 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:13:13.770370 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:13.770329 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:13:31.050618 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:31.050574 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:13:35.666018 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:35.665977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:13:36.423284 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:36.423252 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks"] Jul 06 16:13:36.450940 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:36.450907 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:13:47.970706 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:47.970672 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:13:49.622310 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:49.622278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:13:51.445630 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.445576 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" podUID="29be4850-f11f-42c8-8b2e-5baeb09e4278" containerName="main" containerID="cri-o://63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261" gracePeriod=45 Jul 06 16:13:51.706747 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.706673 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:13:51.799905 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.799869 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-home\") pod \"29be4850-f11f-42c8-8b2e-5baeb09e4278\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " Jul 06 16:13:51.800082 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.799942 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljmwb\" (UniqueName: \"kubernetes.io/projected/29be4850-f11f-42c8-8b2e-5baeb09e4278-kube-api-access-ljmwb\") pod \"29be4850-f11f-42c8-8b2e-5baeb09e4278\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " Jul 06 16:13:51.800082 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800022 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-dshm\") pod \"29be4850-f11f-42c8-8b2e-5baeb09e4278\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " Jul 06 16:13:51.800220 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-model-cache\") pod \"29be4850-f11f-42c8-8b2e-5baeb09e4278\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " Jul 06 16:13:51.800220 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800175 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-home" (OuterVolumeSpecName: "home") pod "29be4850-f11f-42c8-8b2e-5baeb09e4278" (UID: "29be4850-f11f-42c8-8b2e-5baeb09e4278"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:13:51.800331 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800231 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29be4850-f11f-42c8-8b2e-5baeb09e4278-tls-certs\") pod \"29be4850-f11f-42c8-8b2e-5baeb09e4278\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " Jul 06 16:13:51.800331 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800255 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-tmp-dir\") pod \"29be4850-f11f-42c8-8b2e-5baeb09e4278\" (UID: \"29be4850-f11f-42c8-8b2e-5baeb09e4278\") " Jul 06 16:13:51.800438 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800393 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-model-cache" (OuterVolumeSpecName: "model-cache") pod "29be4850-f11f-42c8-8b2e-5baeb09e4278" (UID: "29be4850-f11f-42c8-8b2e-5baeb09e4278"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:13:51.800580 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800550 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29be4850-f11f-42c8-8b2e-5baeb09e4278" (UID: "29be4850-f11f-42c8-8b2e-5baeb09e4278"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:13:51.800580 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800573 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-model-cache\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:13:51.800705 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.800590 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-home\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:13:51.802405 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.802381 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-dshm" (OuterVolumeSpecName: "dshm") pod "29be4850-f11f-42c8-8b2e-5baeb09e4278" (UID: "29be4850-f11f-42c8-8b2e-5baeb09e4278"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:13:51.802472 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.802403 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29be4850-f11f-42c8-8b2e-5baeb09e4278-kube-api-access-ljmwb" (OuterVolumeSpecName: "kube-api-access-ljmwb") pod "29be4850-f11f-42c8-8b2e-5baeb09e4278" (UID: "29be4850-f11f-42c8-8b2e-5baeb09e4278"). InnerVolumeSpecName "kube-api-access-ljmwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:13:51.802472 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.802433 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29be4850-f11f-42c8-8b2e-5baeb09e4278-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29be4850-f11f-42c8-8b2e-5baeb09e4278" (UID: "29be4850-f11f-42c8-8b2e-5baeb09e4278"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:13:51.901527 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.901491 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-dshm\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:13:51.901527 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.901522 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29be4850-f11f-42c8-8b2e-5baeb09e4278-tls-certs\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:13:51.901739 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.901555 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29be4850-f11f-42c8-8b2e-5baeb09e4278-tmp-dir\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:13:51.901739 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:51.901565 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljmwb\" (UniqueName: \"kubernetes.io/projected/29be4850-f11f-42c8-8b2e-5baeb09e4278-kube-api-access-ljmwb\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:13:52.130265 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.130224 2575 generic.go:356] "Generic (PLEG): container finished" podID="29be4850-f11f-42c8-8b2e-5baeb09e4278" containerID="63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261" exitCode=0 Jul 06 16:13:52.130474 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.130277 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" Jul 06 16:13:52.130474 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.130307 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" event={"ID":"29be4850-f11f-42c8-8b2e-5baeb09e4278","Type":"ContainerDied","Data":"63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261"} Jul 06 16:13:52.130474 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.130352 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks" event={"ID":"29be4850-f11f-42c8-8b2e-5baeb09e4278","Type":"ContainerDied","Data":"6b2847808007e933105c2b8a8906053cdb712fae3a66f53ae37ca47178c42558"} Jul 06 16:13:52.130474 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.130370 2575 scope.go:117] "RemoveContainer" containerID="63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261" Jul 06 16:13:52.139677 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.139658 2575 scope.go:117] "RemoveContainer" containerID="63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261" Jul 06 16:13:52.140006 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:13:52.139977 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261\": container with ID starting with 63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261 not found: ID does not exist" containerID="63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261" Jul 06 16:13:52.140086 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.140024 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261"} err="failed to get container status \"63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261\": rpc error: code = NotFound desc = could not find container \"63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261\": container with ID starting with 63f8202d2bd69e97a4d4de04949b0b49f1fd26264b7934d3fc738f7f5d5a6261 not found: ID does not exist" Jul 06 16:13:52.152326 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.152293 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks"] Jul 06 16:13:52.156261 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:52.156235 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/auth-test-model-b747bf-kserve-84c96cfb67-pkjks"] Jul 06 16:13:53.603588 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:53.603555 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29be4850-f11f-42c8-8b2e-5baeb09e4278" path="/var/lib/kubelet/pods/29be4850-f11f-42c8-8b2e-5baeb09e4278/volumes" Jul 06 16:13:59.660656 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:13:59.660612 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:14:08.340735 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:08.340692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:14:40.035536 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.035437 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb"] Jul 06 16:14:40.036096 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.035993 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29be4850-f11f-42c8-8b2e-5baeb09e4278" containerName="main" Jul 06 16:14:40.036096 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.036013 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="29be4850-f11f-42c8-8b2e-5baeb09e4278" containerName="main" Jul 06 16:14:40.036248 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.036108 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="29be4850-f11f-42c8-8b2e-5baeb09e4278" containerName="main" Jul 06 16:14:40.039452 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.039428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.042029 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.041970 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"openshift-service-ca.crt\"" Jul 06 16:14:40.042899 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.042878 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"default-dockercfg-qw6cg\"" Jul 06 16:14:40.043013 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.042895 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"kube-root-ca.crt\"" Jul 06 16:14:40.043013 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.042981 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b747bf\"/\"sub-test-model-b747bf-kserve-self-signed-certs\"" Jul 06 16:14:40.049835 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.049810 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb"] Jul 06 16:14:40.069236 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.069209 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:14:40.142664 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.142627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877985f7-ff65-4b7b-9c20-89790b83e712-tls-certs\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.142664 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.142667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-model-cache\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.142876 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.142747 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-home\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.142876 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.142835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-tmp-dir\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.142947 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.142882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-dshm\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.142947 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.142900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fflz\" (UniqueName: \"kubernetes.io/projected/877985f7-ff65-4b7b-9c20-89790b83e712-kube-api-access-9fflz\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.243948 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.243907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-home\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244141 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.243970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-tmp-dir\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244141 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.243995 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-dshm\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244141 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.244010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fflz\" (UniqueName: \"kubernetes.io/projected/877985f7-ff65-4b7b-9c20-89790b83e712-kube-api-access-9fflz\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244141 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.244041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877985f7-ff65-4b7b-9c20-89790b83e712-tls-certs\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244141 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.244060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-model-cache\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244454 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.244406 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-home\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244501 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.244456 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-tmp-dir\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.244537 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.244515 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-model-cache\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.246440 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.246416 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-dshm\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.246660 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.246640 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877985f7-ff65-4b7b-9c20-89790b83e712-tls-certs\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.255498 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.255474 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fflz\" (UniqueName: \"kubernetes.io/projected/877985f7-ff65-4b7b-9c20-89790b83e712-kube-api-access-9fflz\") pod \"sub-test-model-b747bf-kserve-64f4cdc5-shvzb\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.349538 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.349442 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:40.489830 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:40.489804 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb"] Jul 06 16:14:40.491972 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:14:40.491945 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod877985f7_ff65_4b7b_9c20_89790b83e712.slice/crio-b2be054d22fa37fc56f7d133ecfeb4a5ea72a6b99485ba3ad53bb7f68863b380 WatchSource:0}: Error finding container b2be054d22fa37fc56f7d133ecfeb4a5ea72a6b99485ba3ad53bb7f68863b380: Status 404 returned error can't find the container with id b2be054d22fa37fc56f7d133ecfeb4a5ea72a6b99485ba3ad53bb7f68863b380 Jul 06 16:14:41.359082 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:41.359043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" event={"ID":"877985f7-ff65-4b7b-9c20-89790b83e712","Type":"ContainerStarted","Data":"8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462"} Jul 06 16:14:41.359082 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:41.359086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" event={"ID":"877985f7-ff65-4b7b-9c20-89790b83e712","Type":"ContainerStarted","Data":"b2be054d22fa37fc56f7d133ecfeb4a5ea72a6b99485ba3ad53bb7f68863b380"} Jul 06 16:14:41.384048 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:41.383994 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" podStartSLOduration=1.383978 podStartE2EDuration="1.383978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:14:41.381871554 +0000 UTC m=+2450.316621614" watchObservedRunningTime="2026-07-06 16:14:41.383978 +0000 UTC m=+2450.318728047" Jul 06 16:14:44.260342 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:44.260308 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:14:50.350242 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:50.350195 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:50.350625 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:50.350278 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:50.362689 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:50.362656 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:50.413068 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:50.413033 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:14:53.385309 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:53.385265 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:14:53.507235 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:14:53.507204 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:15:12.135631 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:12.135590 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:15:30.041600 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:30.041562 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:15:34.560152 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:34.560114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:15:35.346210 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:35.346157 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb"] Jul 06 16:15:35.361418 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:35.361373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:15:38.463153 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:38.463110 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:15:38.560928 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:38.560893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:15:45.884065 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:45.883976 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:15:47.359138 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:47.359093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:15:47.458506 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:47.458476 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:15:50.366922 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.366881 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" podUID="877985f7-ff65-4b7b-9c20-89790b83e712" containerName="main" containerID="cri-o://8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462" gracePeriod=45 Jul 06 16:15:50.402981 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.402945 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" podUID="877985f7-ff65-4b7b-9c20-89790b83e712" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": dial tcp 10.133.0.37:8000: connect: connection refused" Jul 06 16:15:50.613932 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.613909 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:15:50.676481 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.676385 2575 generic.go:356] "Generic (PLEG): container finished" podID="877985f7-ff65-4b7b-9c20-89790b83e712" containerID="8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462" exitCode=0 Jul 06 16:15:50.676481 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.676428 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" Jul 06 16:15:50.676481 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.676448 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" event={"ID":"877985f7-ff65-4b7b-9c20-89790b83e712","Type":"ContainerDied","Data":"8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462"} Jul 06 16:15:50.676729 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.676487 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb" event={"ID":"877985f7-ff65-4b7b-9c20-89790b83e712","Type":"ContainerDied","Data":"b2be054d22fa37fc56f7d133ecfeb4a5ea72a6b99485ba3ad53bb7f68863b380"} Jul 06 16:15:50.676729 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.676510 2575 scope.go:117] "RemoveContainer" containerID="8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462" Jul 06 16:15:50.685101 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.685082 2575 scope.go:117] "RemoveContainer" containerID="8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462" Jul 06 16:15:50.685369 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:15:50.685351 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462\": container with ID starting with 8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462 not found: ID does not exist" containerID="8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462" Jul 06 16:15:50.685468 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.685380 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462"} err="failed to get container status \"8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462\": rpc error: code = NotFound desc = could not find container \"8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462\": container with ID starting with 8e96822ab1415af39d0a387c2ab17d2bf608ca543bc667ffcfc38b5aae415462 not found: ID does not exist" Jul 06 16:15:50.753900 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.753872 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-model-cache\") pod \"877985f7-ff65-4b7b-9c20-89790b83e712\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " Jul 06 16:15:50.754020 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.753919 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877985f7-ff65-4b7b-9c20-89790b83e712-tls-certs\") pod \"877985f7-ff65-4b7b-9c20-89790b83e712\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " Jul 06 16:15:50.754020 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754000 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fflz\" (UniqueName: \"kubernetes.io/projected/877985f7-ff65-4b7b-9c20-89790b83e712-kube-api-access-9fflz\") pod \"877985f7-ff65-4b7b-9c20-89790b83e712\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " Jul 06 16:15:50.754133 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754045 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-home\") pod \"877985f7-ff65-4b7b-9c20-89790b83e712\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " Jul 06 16:15:50.754133 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754069 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-tmp-dir\") pod \"877985f7-ff65-4b7b-9c20-89790b83e712\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " Jul 06 16:15:50.754133 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754106 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-dshm\") pod \"877985f7-ff65-4b7b-9c20-89790b83e712\" (UID: \"877985f7-ff65-4b7b-9c20-89790b83e712\") " Jul 06 16:15:50.754311 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754158 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-model-cache" (OuterVolumeSpecName: "model-cache") pod "877985f7-ff65-4b7b-9c20-89790b83e712" (UID: "877985f7-ff65-4b7b-9c20-89790b83e712"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:15:50.754361 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754348 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-home" (OuterVolumeSpecName: "home") pod "877985f7-ff65-4b7b-9c20-89790b83e712" (UID: "877985f7-ff65-4b7b-9c20-89790b83e712"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:15:50.754408 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754362 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "877985f7-ff65-4b7b-9c20-89790b83e712" (UID: "877985f7-ff65-4b7b-9c20-89790b83e712"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:15:50.754501 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754482 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-home\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:15:50.754571 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754512 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-tmp-dir\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:15:50.754571 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.754531 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-model-cache\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:15:50.756343 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.756318 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877985f7-ff65-4b7b-9c20-89790b83e712-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "877985f7-ff65-4b7b-9c20-89790b83e712" (UID: "877985f7-ff65-4b7b-9c20-89790b83e712"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:15:50.756427 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.756351 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-dshm" (OuterVolumeSpecName: "dshm") pod "877985f7-ff65-4b7b-9c20-89790b83e712" (UID: "877985f7-ff65-4b7b-9c20-89790b83e712"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:15:50.756667 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.756643 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877985f7-ff65-4b7b-9c20-89790b83e712-kube-api-access-9fflz" (OuterVolumeSpecName: "kube-api-access-9fflz") pod "877985f7-ff65-4b7b-9c20-89790b83e712" (UID: "877985f7-ff65-4b7b-9c20-89790b83e712"). InnerVolumeSpecName "kube-api-access-9fflz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:15:50.855420 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.855394 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877985f7-ff65-4b7b-9c20-89790b83e712-tls-certs\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:15:50.855420 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.855415 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fflz\" (UniqueName: \"kubernetes.io/projected/877985f7-ff65-4b7b-9c20-89790b83e712-kube-api-access-9fflz\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:15:50.855420 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.855424 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877985f7-ff65-4b7b-9c20-89790b83e712-dshm\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:15:50.998782 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:50.998752 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb"] Jul 06 16:15:51.002370 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:51.002349 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b747bf/sub-test-model-b747bf-kserve-64f4cdc5-shvzb"] Jul 06 16:15:51.611135 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:15:51.611097 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="877985f7-ff65-4b7b-9c20-89790b83e712" path="/var/lib/kubelet/pods/877985f7-ff65-4b7b-9c20-89790b83e712/volumes" Jul 06 16:16:00.351121 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.351089 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8"] Jul 06 16:16:00.351586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.351456 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877985f7-ff65-4b7b-9c20-89790b83e712" containerName="main" Jul 06 16:16:00.351586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.351467 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="877985f7-ff65-4b7b-9c20-89790b83e712" containerName="main" Jul 06 16:16:00.351586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.351521 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="877985f7-ff65-4b7b-9c20-89790b83e712" containerName="main" Jul 06 16:16:00.354419 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.354403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.358017 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.357991 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"default-dockercfg-fkgd7\"" Jul 06 16:16:00.358186 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.358045 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"rate-test-model-c8a9d1-kserve-self-signed-certs\"" Jul 06 16:16:00.358282 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.358263 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"kube-root-ca.crt\"" Jul 06 16:16:00.358352 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.358312 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c8a9d1\"/\"openshift-service-ca.crt\"" Jul 06 16:16:00.365179 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.365145 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8"] Jul 06 16:16:00.416148 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.416122 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:16:00.435993 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.435970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-model-cache\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.436086 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.436006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-home\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.436086 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.436026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-dshm\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.436155 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.436103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tmp-dir\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.436155 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.436150 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79vl7\" (UniqueName: \"kubernetes.io/projected/182bd6db-5281-408d-8d5d-6eddf6b9faf0-kube-api-access-79vl7\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.436239 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.436220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tls-certs\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.536679 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.536654 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-model-cache\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.536775 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.536691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-home\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.536775 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.536712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-dshm\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.536775 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.536761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tmp-dir\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.536973 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.536801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79vl7\" (UniqueName: \"kubernetes.io/projected/182bd6db-5281-408d-8d5d-6eddf6b9faf0-kube-api-access-79vl7\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.536973 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.536837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tls-certs\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.537073 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.537025 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-home\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.537131 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.537073 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-model-cache\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.537216 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.537188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tmp-dir\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.539187 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.539143 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-dshm\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.539400 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.539384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tls-certs\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.546950 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.546920 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79vl7\" (UniqueName: \"kubernetes.io/projected/182bd6db-5281-408d-8d5d-6eddf6b9faf0-kube-api-access-79vl7\") pod \"rate-test-model-c8a9d1-kserve-74c9d89675-cfws8\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.663950 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.663883 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:00.793633 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:00.793609 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8"] Jul 06 16:16:00.795665 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:16:00.795629 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod182bd6db_5281_408d_8d5d_6eddf6b9faf0.slice/crio-21413baf903a90472dfb3bf4ac1b06c9b6b6997cd2367a90d2552c1dc7612837 WatchSource:0}: Error finding container 21413baf903a90472dfb3bf4ac1b06c9b6b6997cd2367a90d2552c1dc7612837: Status 404 returned error can't find the container with id 21413baf903a90472dfb3bf4ac1b06c9b6b6997cd2367a90d2552c1dc7612837 Jul 06 16:16:01.733540 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:01.733512 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" event={"ID":"182bd6db-5281-408d-8d5d-6eddf6b9faf0","Type":"ContainerStarted","Data":"58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856"} Jul 06 16:16:01.733910 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:01.733547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" event={"ID":"182bd6db-5281-408d-8d5d-6eddf6b9faf0","Type":"ContainerStarted","Data":"21413baf903a90472dfb3bf4ac1b06c9b6b6997cd2367a90d2552c1dc7612837"} Jul 06 16:16:01.752680 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:01.752628 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" podStartSLOduration=1.752611918 podStartE2EDuration="1.752611918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:16:01.751770384 +0000 UTC m=+2530.686520432" watchObservedRunningTime="2026-07-06 16:16:01.752611918 +0000 UTC m=+2530.687361973" Jul 06 16:16:03.595110 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:03.595074 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:16:10.664811 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:10.664782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:10.665275 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:10.664828 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:10.677137 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:10.677113 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:10.785253 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:10.785225 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:16:28.088256 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:28.088219 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:16:32.074219 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:32.074181 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:16:51.145874 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:16:51.145834 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:17:05.273414 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:05.273375 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:17:12.010813 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:12.010733 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8"] Jul 06 16:17:12.085817 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:12.085777 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:14.878853 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:14.878819 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r"] Jul 06 16:17:14.879375 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:14.879066 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" podUID="05749eb4-e78f-422f-aceb-5edb4d1690bf" containerName="maas-api" containerID="cri-o://ce1fb49285ecc03f94d668076255c981061cb188b8ee28285be8cdb22722cc43" gracePeriod=30 Jul 06 16:17:15.067412 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.067345 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:15.083909 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.083875 2575 generic.go:356] "Generic (PLEG): container finished" podID="05749eb4-e78f-422f-aceb-5edb4d1690bf" containerID="ce1fb49285ecc03f94d668076255c981061cb188b8ee28285be8cdb22722cc43" exitCode=0 Jul 06 16:17:15.084053 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.083945 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" event={"ID":"05749eb4-e78f-422f-aceb-5edb4d1690bf","Type":"ContainerDied","Data":"ce1fb49285ecc03f94d668076255c981061cb188b8ee28285be8cdb22722cc43"} Jul 06 16:17:15.128060 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.128034 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:17:15.306115 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.306082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05749eb4-e78f-422f-aceb-5edb4d1690bf-openshift-service-ca\") pod \"05749eb4-e78f-422f-aceb-5edb4d1690bf\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " Jul 06 16:17:15.306321 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.306142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgcmg\" (UniqueName: \"kubernetes.io/projected/05749eb4-e78f-422f-aceb-5edb4d1690bf-kube-api-access-bgcmg\") pod \"05749eb4-e78f-422f-aceb-5edb4d1690bf\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " Jul 06 16:17:15.306321 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.306202 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls\") pod \"05749eb4-e78f-422f-aceb-5edb4d1690bf\" (UID: \"05749eb4-e78f-422f-aceb-5edb4d1690bf\") " Jul 06 16:17:15.306508 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.306484 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05749eb4-e78f-422f-aceb-5edb4d1690bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "05749eb4-e78f-422f-aceb-5edb4d1690bf" (UID: "05749eb4-e78f-422f-aceb-5edb4d1690bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:17:15.308376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.308341 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05749eb4-e78f-422f-aceb-5edb4d1690bf-kube-api-access-bgcmg" (OuterVolumeSpecName: "kube-api-access-bgcmg") pod "05749eb4-e78f-422f-aceb-5edb4d1690bf" (UID: "05749eb4-e78f-422f-aceb-5edb4d1690bf"). InnerVolumeSpecName "kube-api-access-bgcmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:17:15.308526 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.308508 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "05749eb4-e78f-422f-aceb-5edb4d1690bf" (UID: "05749eb4-e78f-422f-aceb-5edb4d1690bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:17:15.407063 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.407026 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05749eb4-e78f-422f-aceb-5edb4d1690bf-openshift-service-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:15.407063 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.407055 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgcmg\" (UniqueName: \"kubernetes.io/projected/05749eb4-e78f-422f-aceb-5edb4d1690bf-kube-api-access-bgcmg\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:15.407063 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:15.407066 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05749eb4-e78f-422f-aceb-5edb4d1690bf-maas-api-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:16.090096 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:16.090063 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" Jul 06 16:17:16.090096 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:16.090074 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r" event={"ID":"05749eb4-e78f-422f-aceb-5edb4d1690bf","Type":"ContainerDied","Data":"4ac24e62757edb992c9aa349537a86f05277c8ea4888b925db8d86aad6e16ba0"} Jul 06 16:17:16.090571 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:16.090128 2575 scope.go:117] "RemoveContainer" containerID="ce1fb49285ecc03f94d668076255c981061cb188b8ee28285be8cdb22722cc43" Jul 06 16:17:16.106732 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:16.106705 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r"] Jul 06 16:17:16.112671 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:16.112649 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-c8a9d1-7f95d9f7b4-bpg9r"] Jul 06 16:17:17.603254 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:17.603224 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05749eb4-e78f-422f-aceb-5edb4d1690bf" path="/var/lib/kubelet/pods/05749eb4-e78f-422f-aceb-5edb4d1690bf/volumes" Jul 06 16:17:18.036434 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:18.036396 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:17:19.675107 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:19.675072 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:27.030807 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.030752 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" podUID="182bd6db-5281-408d-8d5d-6eddf6b9faf0" containerName="main" containerID="cri-o://58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856" gracePeriod=45 Jul 06 16:17:27.288502 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.288437 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:17:27.316536 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316494 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-model-cache\") pod \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " Jul 06 16:17:27.316724 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316589 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tls-certs\") pod \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " Jul 06 16:17:27.316724 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-home\") pod \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " Jul 06 16:17:27.316724 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tmp-dir\") pod \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " Jul 06 16:17:27.316892 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316727 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79vl7\" (UniqueName: \"kubernetes.io/projected/182bd6db-5281-408d-8d5d-6eddf6b9faf0-kube-api-access-79vl7\") pod \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " Jul 06 16:17:27.316892 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316756 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-dshm\") pod \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\" (UID: \"182bd6db-5281-408d-8d5d-6eddf6b9faf0\") " Jul 06 16:17:27.316892 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316852 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-model-cache" (OuterVolumeSpecName: "model-cache") pod "182bd6db-5281-408d-8d5d-6eddf6b9faf0" (UID: "182bd6db-5281-408d-8d5d-6eddf6b9faf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:17:27.317045 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.316887 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-home" (OuterVolumeSpecName: "home") pod "182bd6db-5281-408d-8d5d-6eddf6b9faf0" (UID: "182bd6db-5281-408d-8d5d-6eddf6b9faf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:17:27.317045 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.317008 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "182bd6db-5281-408d-8d5d-6eddf6b9faf0" (UID: "182bd6db-5281-408d-8d5d-6eddf6b9faf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:17:27.317178 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.317140 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-home\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:27.317239 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.317222 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tmp-dir\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:27.317278 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.317244 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-model-cache\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:27.319250 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.319218 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-dshm" (OuterVolumeSpecName: "dshm") pod "182bd6db-5281-408d-8d5d-6eddf6b9faf0" (UID: "182bd6db-5281-408d-8d5d-6eddf6b9faf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:17:27.319250 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.319229 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182bd6db-5281-408d-8d5d-6eddf6b9faf0-kube-api-access-79vl7" (OuterVolumeSpecName: "kube-api-access-79vl7") pod "182bd6db-5281-408d-8d5d-6eddf6b9faf0" (UID: "182bd6db-5281-408d-8d5d-6eddf6b9faf0"). InnerVolumeSpecName "kube-api-access-79vl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:17:27.319417 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.319348 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "182bd6db-5281-408d-8d5d-6eddf6b9faf0" (UID: "182bd6db-5281-408d-8d5d-6eddf6b9faf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:17:27.418358 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.418308 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/182bd6db-5281-408d-8d5d-6eddf6b9faf0-tls-certs\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:27.418358 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.418350 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79vl7\" (UniqueName: \"kubernetes.io/projected/182bd6db-5281-408d-8d5d-6eddf6b9faf0-kube-api-access-79vl7\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:27.418358 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:27.418360 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/182bd6db-5281-408d-8d5d-6eddf6b9faf0-dshm\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:28.146155 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.146116 2575 generic.go:356] "Generic (PLEG): container finished" podID="182bd6db-5281-408d-8d5d-6eddf6b9faf0" containerID="58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856" exitCode=0 Jul 06 16:17:28.146586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.146190 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" Jul 06 16:17:28.146586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.146206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" event={"ID":"182bd6db-5281-408d-8d5d-6eddf6b9faf0","Type":"ContainerDied","Data":"58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856"} Jul 06 16:17:28.146586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.146248 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8" event={"ID":"182bd6db-5281-408d-8d5d-6eddf6b9faf0","Type":"ContainerDied","Data":"21413baf903a90472dfb3bf4ac1b06c9b6b6997cd2367a90d2552c1dc7612837"} Jul 06 16:17:28.146586 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.146264 2575 scope.go:117] "RemoveContainer" containerID="58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856" Jul 06 16:17:28.154439 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.154423 2575 scope.go:117] "RemoveContainer" containerID="58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856" Jul 06 16:17:28.154706 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:17:28.154687 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856\": container with ID starting with 58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856 not found: ID does not exist" containerID="58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856" Jul 06 16:17:28.154750 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.154714 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856"} err="failed to get container status \"58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856\": rpc error: code = NotFound desc = could not find container \"58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856\": container with ID starting with 58169ee5247004e3286497fabeedd7e2996c30b3971c33584dbcf2f90b44d856 not found: ID does not exist" Jul 06 16:17:28.164516 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.164492 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8"] Jul 06 16:17:28.170694 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:28.170672 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c8a9d1/rate-test-model-c8a9d1-kserve-74c9d89675-cfws8"] Jul 06 16:17:29.603717 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:29.603673 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="182bd6db-5281-408d-8d5d-6eddf6b9faf0" path="/var/lib/kubelet/pods/182bd6db-5281-408d-8d5d-6eddf6b9faf0/volumes" Jul 06 16:17:57.787622 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:57.787589 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:58.062909 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:58.062829 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:59.009012 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.008977 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:59.115005 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.114972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:59.249637 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.249606 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9"] Jul 06 16:17:59.249861 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.249813 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" podUID="9a408186-f3c2-4925-8aa8-38be943e3596" containerName="maas-api" containerID="cri-o://4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16" gracePeriod=30 Jul 06 16:17:59.287562 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.287530 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:17:59.495523 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.495498 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:17:59.600504 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.600473 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a408186-f3c2-4925-8aa8-38be943e3596-maas-api-tls\") pod \"9a408186-f3c2-4925-8aa8-38be943e3596\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " Jul 06 16:17:59.600504 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.600509 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a408186-f3c2-4925-8aa8-38be943e3596-openshift-service-ca\") pod \"9a408186-f3c2-4925-8aa8-38be943e3596\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " Jul 06 16:17:59.600767 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.600645 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6shjv\" (UniqueName: \"kubernetes.io/projected/9a408186-f3c2-4925-8aa8-38be943e3596-kube-api-access-6shjv\") pod \"9a408186-f3c2-4925-8aa8-38be943e3596\" (UID: \"9a408186-f3c2-4925-8aa8-38be943e3596\") " Jul 06 16:17:59.600904 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.600882 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a408186-f3c2-4925-8aa8-38be943e3596-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a408186-f3c2-4925-8aa8-38be943e3596" (UID: "9a408186-f3c2-4925-8aa8-38be943e3596"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:17:59.602800 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.602775 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a408186-f3c2-4925-8aa8-38be943e3596-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a408186-f3c2-4925-8aa8-38be943e3596" (UID: "9a408186-f3c2-4925-8aa8-38be943e3596"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:17:59.603004 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.602985 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a408186-f3c2-4925-8aa8-38be943e3596-kube-api-access-6shjv" (OuterVolumeSpecName: "kube-api-access-6shjv") pod "9a408186-f3c2-4925-8aa8-38be943e3596" (UID: "9a408186-f3c2-4925-8aa8-38be943e3596"). InnerVolumeSpecName "kube-api-access-6shjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:17:59.702241 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.702202 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6shjv\" (UniqueName: \"kubernetes.io/projected/9a408186-f3c2-4925-8aa8-38be943e3596-kube-api-access-6shjv\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:59.702241 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.702237 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a408186-f3c2-4925-8aa8-38be943e3596-maas-api-tls\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:17:59.702241 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:17:59.702247 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a408186-f3c2-4925-8aa8-38be943e3596-openshift-service-ca\") on node \"ip-10-0-131-241.ec2.internal\" DevicePath \"\"" Jul 06 16:18:00.302768 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.302734 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a408186-f3c2-4925-8aa8-38be943e3596" containerID="4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16" exitCode=0 Jul 06 16:18:00.303209 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.302777 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" Jul 06 16:18:00.303209 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.302813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" event={"ID":"9a408186-f3c2-4925-8aa8-38be943e3596","Type":"ContainerDied","Data":"4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16"} Jul 06 16:18:00.303209 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.302869 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9" event={"ID":"9a408186-f3c2-4925-8aa8-38be943e3596","Type":"ContainerDied","Data":"e9e6f18b24b64e217735c729718e50a2d0c8274750523431b4ea62e37d1fc6a7"} Jul 06 16:18:00.303209 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.302898 2575 scope.go:117] "RemoveContainer" containerID="4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16" Jul 06 16:18:00.311343 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.311320 2575 scope.go:117] "RemoveContainer" containerID="4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16" Jul 06 16:18:00.311595 ip-10-0-131-241 kubenswrapper[2575]: E0706 16:18:00.311576 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16\": container with ID starting with 4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16 not found: ID does not exist" containerID="4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16" Jul 06 16:18:00.311656 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.311601 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16"} err="failed to get container status \"4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16\": rpc error: code = NotFound desc = could not find container \"4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16\": container with ID starting with 4241c90fe27d91c618fcf274b720323bd2423a13642fed74110c3d0c946eee16 not found: ID does not exist" Jul 06 16:18:00.325423 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.325371 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9"] Jul 06 16:18:00.330977 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:00.330946 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-b747bf-67d8ccb4b5-sxwx9"] Jul 06 16:18:01.604327 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:01.604295 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a408186-f3c2-4925-8aa8-38be943e3596" path="/var/lib/kubelet/pods/9a408186-f3c2-4925-8aa8-38be943e3596/volumes" Jul 06 16:18:02.064670 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:02.064634 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gjbhb"] Jul 06 16:18:06.459723 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:06.459686 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:06.530915 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:06.530839 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:06.823394 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:06.823362 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:07.067710 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:07.067661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:10.115445 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:10.115411 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:13.418295 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:13.418260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:13.448220 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:13.448197 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-f6b99_3951674d-a6dd-4297-a41f-41b48426e583/copy-manifests/0.log" Jul 06 16:18:13.456512 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:13.456494 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-f6b99_3951674d-a6dd-4297-a41f-41b48426e583/manager/0.log" Jul 06 16:18:13.516007 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:13.515972 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:15.316206 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:15.316176 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zslmt_980a6f0f-5f46-4408-814e-ff7f1999187e/manager/0.log" Jul 06 16:18:15.665493 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:15.665417 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bhqqs_0770cf65-eeb9-4e1e-91fa-98f9823dfdb5/manager/0.log" Jul 06 16:18:15.773581 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:15.773556 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gjbhb_db951259-31f9-4dce-b5ed-595094f981a3/limitador/0.log" Jul 06 16:18:16.359401 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:16.359355 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5t4zg_0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9/discovery/0.log" Jul 06 16:18:17.911211 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:17.911180 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-b7js8_9aac0d1b-ad7a-40bd-9405-4baee1e9b1ad/main/0.log" Jul 06 16:18:22.179996 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:22.179956 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:22.277975 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:22.277947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" limit=1024 Jul 06 16:18:26.099260 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:26.099228 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jtj2c_2b0126b9-095c-4bbb-a0b6-a9c2e9359993/global-pull-secret-syncer/0.log" Jul 06 16:18:26.192673 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:26.192642 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8nfwc_a990e587-21da-439e-bc4c-7e785b8e6be9/konnectivity-agent/0.log" Jul 06 16:18:26.332427 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:26.332397 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-241.ec2.internal_c079fdaed824bcba1a1cdfbf52df0d2c/haproxy/0.log" Jul 06 16:18:31.678651 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:31.678566 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zslmt_980a6f0f-5f46-4408-814e-ff7f1999187e/manager/0.log" Jul 06 16:18:31.883115 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:31.883077 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bhqqs_0770cf65-eeb9-4e1e-91fa-98f9823dfdb5/manager/0.log" Jul 06 16:18:31.916456 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:31.916427 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gjbhb_db951259-31f9-4dce-b5ed-595094f981a3/limitador/0.log" Jul 06 16:18:33.683531 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:33.683495 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69/alertmanager/0.log" Jul 06 16:18:33.728103 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:33.728074 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69/config-reloader/0.log" Jul 06 16:18:33.779544 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:33.779519 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69/kube-rbac-proxy-web/0.log" Jul 06 16:18:33.842841 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:33.842765 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69/kube-rbac-proxy/0.log" Jul 06 16:18:33.881820 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:33.881796 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69/kube-rbac-proxy-metric/0.log" Jul 06 16:18:33.912596 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:33.912567 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69/prom-label-proxy/0.log" Jul 06 16:18:33.936595 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:33.936568 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2d4aa4fb-d496-4fc8-8ee0-64ed3e5b0e69/init-config-reloader/0.log" Jul 06 16:18:34.275446 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.275365 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-564pg_0f63d001-e301-4d65-b409-cda9932c4ea3/node-exporter/0.log" Jul 06 16:18:34.303633 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.303602 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-564pg_0f63d001-e301-4d65-b409-cda9932c4ea3/kube-rbac-proxy/0.log" Jul 06 16:18:34.332820 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.332798 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-564pg_0f63d001-e301-4d65-b409-cda9932c4ea3/init-textfile/0.log" Jul 06 16:18:34.709062 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.709034 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-66zd9_7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb/kube-rbac-proxy-main/0.log" Jul 06 16:18:34.745597 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.745567 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-66zd9_7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb/kube-rbac-proxy-self/0.log" Jul 06 16:18:34.780465 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.780441 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-66zd9_7e0626e8-8b36-40b7-8d1a-3a3626d4bfdb/openshift-state-metrics/0.log" Jul 06 16:18:34.828039 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.828013 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b96d5d40-01bd-4492-bd78-9219c3e1cf13/prometheus/0.log" Jul 06 16:18:34.853005 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.852981 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b96d5d40-01bd-4492-bd78-9219c3e1cf13/config-reloader/0.log" Jul 06 16:18:34.895686 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.895660 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b96d5d40-01bd-4492-bd78-9219c3e1cf13/thanos-sidecar/0.log" Jul 06 16:18:34.928567 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.928537 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b96d5d40-01bd-4492-bd78-9219c3e1cf13/kube-rbac-proxy-web/0.log" Jul 06 16:18:34.964370 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:34.964299 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b96d5d40-01bd-4492-bd78-9219c3e1cf13/kube-rbac-proxy/0.log" Jul 06 16:18:35.001302 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:35.001274 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b96d5d40-01bd-4492-bd78-9219c3e1cf13/kube-rbac-proxy-thanos/0.log" Jul 06 16:18:35.043045 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:35.043022 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b96d5d40-01bd-4492-bd78-9219c3e1cf13/init-config-reloader/0.log" Jul 06 16:18:35.196240 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:35.196208 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64dfc4c9-5bjwk_f29da444-c54c-44aa-876b-f02a6c5f05d9/telemeter-client/0.log" Jul 06 16:18:35.259475 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:35.259398 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64dfc4c9-5bjwk_f29da444-c54c-44aa-876b-f02a6c5f05d9/reload/0.log" Jul 06 16:18:35.317362 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:35.317332 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-64dfc4c9-5bjwk_f29da444-c54c-44aa-876b-f02a6c5f05d9/kube-rbac-proxy/0.log" Jul 06 16:18:36.427360 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427329 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v"] Jul 06 16:18:36.427748 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427666 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a408186-f3c2-4925-8aa8-38be943e3596" containerName="maas-api" Jul 06 16:18:36.427748 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427678 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a408186-f3c2-4925-8aa8-38be943e3596" containerName="maas-api" Jul 06 16:18:36.427748 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427692 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="182bd6db-5281-408d-8d5d-6eddf6b9faf0" containerName="main" Jul 06 16:18:36.427748 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427697 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="182bd6db-5281-408d-8d5d-6eddf6b9faf0" containerName="main" Jul 06 16:18:36.427748 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427715 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05749eb4-e78f-422f-aceb-5edb4d1690bf" containerName="maas-api" Jul 06 16:18:36.427748 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427721 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="05749eb4-e78f-422f-aceb-5edb4d1690bf" containerName="maas-api" Jul 06 16:18:36.427975 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427776 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="182bd6db-5281-408d-8d5d-6eddf6b9faf0" containerName="main" Jul 06 16:18:36.427975 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427783 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="05749eb4-e78f-422f-aceb-5edb4d1690bf" containerName="maas-api" Jul 06 16:18:36.427975 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.427788 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a408186-f3c2-4925-8aa8-38be943e3596" containerName="maas-api" Jul 06 16:18:36.432116 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.432085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.435149 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.435123 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z6r42\"/\"default-dockercfg-mwdkp\"" Jul 06 16:18:36.435304 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.435186 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6r42\"/\"openshift-service-ca.crt\"" Jul 06 16:18:36.436076 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.436058 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6r42\"/\"kube-root-ca.crt\"" Jul 06 16:18:36.452872 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.452839 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v"] Jul 06 16:18:36.528192 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.528125 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-proc\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.528368 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.528265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-podres\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.528368 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.528305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-lib-modules\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.528368 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.528336 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dxt6\" (UniqueName: \"kubernetes.io/projected/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-kube-api-access-6dxt6\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.528477 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.528417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-sys\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629044 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-podres\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629044 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-lib-modules\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dxt6\" (UniqueName: \"kubernetes.io/projected/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-kube-api-access-6dxt6\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-sys\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-proc\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629237 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-podres\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629265 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-lib-modules\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629290 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-sys\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.629376 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.629300 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-proc\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.640661 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.640629 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dxt6\" (UniqueName: \"kubernetes.io/projected/bd24ca94-9da2-4809-90bd-ba0ce8bdcb33-kube-api-access-6dxt6\") pod \"perf-node-gather-daemonset-fd62v\" (UID: \"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33\") " pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.744149 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.744050 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:36.904139 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:36.904113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v"] Jul 06 16:18:36.906822 ip-10-0-131-241 kubenswrapper[2575]: W0706 16:18:36.906794 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbd24ca94_9da2_4809_90bd_ba0ce8bdcb33.slice/crio-36cda1acaecf4fdc377b1f3109b324d0e1f02578e8dc1435e4c41589e08a05bc WatchSource:0}: Error finding container 36cda1acaecf4fdc377b1f3109b324d0e1f02578e8dc1435e4c41589e08a05bc: Status 404 returned error can't find the container with id 36cda1acaecf4fdc377b1f3109b324d0e1f02578e8dc1435e4c41589e08a05bc Jul 06 16:18:37.476211 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:37.476158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" event={"ID":"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33","Type":"ContainerStarted","Data":"48680b49b2dfb8fa2db9d0cb2d835203f13dd9136fbad03c512e64e6f7fbbea7"} Jul 06 16:18:37.476687 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:37.476232 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" event={"ID":"bd24ca94-9da2-4809-90bd-ba0ce8bdcb33","Type":"ContainerStarted","Data":"36cda1acaecf4fdc377b1f3109b324d0e1f02578e8dc1435e4c41589e08a05bc"} Jul 06 16:18:37.476687 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:37.476260 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:37.504182 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:37.504118 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" podStartSLOduration=1.50410185 podStartE2EDuration="1.50410185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:18:37.502742519 +0000 UTC m=+2686.437492579" watchObservedRunningTime="2026-07-06 16:18:37.50410185 +0000 UTC m=+2686.438851896" Jul 06 16:18:38.253523 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:38.253491 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-kj88m_304631ed-a1bd-4a6e-88bd-d92b8fdbeb48/download-server/0.log" Jul 06 16:18:40.100614 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:40.100585 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4sbcx_2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52/dns/0.log" Jul 06 16:18:40.128211 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:40.128184 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4sbcx_2ec270e9-050d-4d6e-b2aa-9f6b7b4b2f52/kube-rbac-proxy/0.log" Jul 06 16:18:40.320919 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:40.320895 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-spgk5_fbcf2182-b539-45cd-bc1b-9fa31f2f5b88/dns-node-resolver/0.log" Jul 06 16:18:40.944284 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:40.944248 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q9xfl_c7f213b1-bd93-40d0-8ffe-71b3cb4dcd16/node-ca/0.log" Jul 06 16:18:42.197499 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:42.197424 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5t4zg_0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9/discovery/0.log" Jul 06 16:18:43.184409 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:43.184376 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mzkwj_3352ee41-9ed0-4b96-942d-6f9e18f5c10b/serve-healthcheck-canary/0.log" Jul 06 16:18:43.492175 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:43.492095 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z6r42/perf-node-gather-daemonset-fd62v" Jul 06 16:18:44.016182 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:44.016133 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s7d97_b7c5c181-6f88-4d46-b7d9-bdf86ca0acef/kube-rbac-proxy/0.log" Jul 06 16:18:44.050836 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:44.050802 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s7d97_b7c5c181-6f88-4d46-b7d9-bdf86ca0acef/exporter/0.log" Jul 06 16:18:44.088040 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:44.088015 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s7d97_b7c5c181-6f88-4d46-b7d9-bdf86ca0acef/extractor/0.log" Jul 06 16:18:47.027910 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:47.027826 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5t4zg_0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9/discovery/0.log" Jul 06 16:18:47.505770 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:47.505730 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-f6b99_3951674d-a6dd-4297-a41f-41b48426e583/manager/0.log" Jul 06 16:18:47.545555 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:47.545527 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-749df7788b-f6b99_3951674d-a6dd-4297-a41f-41b48426e583/copy-manifests/0.log" Jul 06 16:18:48.702430 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:48.702342 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5t4zg_0b7a9ceb-0fbf-41f1-870e-f7b0c19361c9/discovery/0.log" Jul 06 16:18:49.204938 ip-10-0-131-241 kubenswrapper[2575]: I0706 16:18:49.204906 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-544dccfdb4-fzxlw_9f76fae5-6ded-479a-a984-792b7d5eb430/manager/0.log"