Jul 12 19:20:14.099358 ip-10-0-141-249 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 12 19:20:14.099369 ip-10-0-141-249 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 12 19:20:14.099377 ip-10-0-141-249 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 12 19:20:14.099585 ip-10-0-141-249 systemd[1]: Failed to start Kubernetes Kubelet. Jul 12 19:20:23.411149 ip-10-0-141-249 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 12 19:20:23.411162 ip-10-0-141-249 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7b039ae49ed74db884f55a07a1fbe086 -- Jul 12 19:21:51.183525 ip-10-0-141-249 systemd[1]: Starting Kubernetes Kubelet... Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.630072 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634742 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634754 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634757 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634761 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634765 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634767 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634770 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634773 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:21:51.973827 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634775 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634778 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634781 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634783 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634791 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634793 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634796 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634799 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634801 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634803 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634806 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634809 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634811 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634815 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634817 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634820 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634822 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634825 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634828 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634831 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:21:51.987258 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634833 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:21:51.713842 ip-10-0-141-249 systemd[1]: Started Kubernetes Kubelet. Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634836 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634839 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634841 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634843 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634846 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634849 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634852 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634855 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634857 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634860 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634863 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634866 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634868 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634871 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634873 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634876 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634878 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634881 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634883 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:21:51.988597 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634886 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634888 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634891 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634893 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634897 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634899 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634902 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634905 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634908 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634913 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634916 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634920 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634924 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634926 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634929 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634933 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634935 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634938 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:21:51.989624 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634941 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634943 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634946 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634948 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634950 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634955 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634959 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634961 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634964 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634967 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634969 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634972 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634974 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634976 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634979 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634981 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634984 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634986 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634988 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634991 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:21:51.990480 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634993 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.634997 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635000 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635002 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635005 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635007 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635010 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635012 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635015 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635017 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635019 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635023 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635025 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635727 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635734 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635737 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635740 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635743 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635747 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635750 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:21:51.991383 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635753 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635755 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635758 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635762 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635766 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635770 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635772 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635775 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635778 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635780 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635783 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635786 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635789 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635793 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635795 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635798 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635800 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635803 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635807 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635809 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:21:51.992255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635812 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635814 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635817 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635819 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635822 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635824 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635827 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635830 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635833 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635835 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635838 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635841 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635845 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635849 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635852 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635854 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635857 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635859 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635861 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:21:51.993548 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635866 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635869 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635871 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635874 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635876 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635879 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635881 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635884 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635887 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635889 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635892 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635895 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635897 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635900 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635903 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635905 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635908 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635910 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635913 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635916 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:21:51.994398 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635919 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635921 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635924 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635927 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635929 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635932 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635935 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635937 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635939 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635942 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635944 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635947 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635951 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635955 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635958 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635961 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635964 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635967 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635969 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:21:51.995486 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635972 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635974 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635977 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635980 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635982 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635988 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635991 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635993 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635996 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.635998 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636001 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636003 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636006 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636008 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636417 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636423 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636426 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636430 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636433 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636436 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:21:51.996403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636438 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636441 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636443 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636446 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636449 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636454 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636457 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636460 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636463 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636466 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636469 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636472 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636474 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636477 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636480 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636483 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636486 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636490 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636492 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:21:51.997661 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636495 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636499 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636502 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636505 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636507 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636510 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636513 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636516 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636519 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636521 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636524 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636527 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636529 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636532 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636534 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636537 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636539 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636542 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636545 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:21:51.998520 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636548 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636552 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636556 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636559 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636562 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636564 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636567 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636569 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636572 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636574 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636577 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636581 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636583 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636586 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636589 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636591 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636594 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636596 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636599 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636601 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:21:51.999601 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636604 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636606 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636609 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636611 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636613 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636616 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636633 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636636 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636640 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636644 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636646 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636651 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636654 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636657 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636660 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636663 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636665 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636668 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636671 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636674 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:21:52.000667 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636677 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636680 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636683 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636687 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636690 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636692 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636695 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636698 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636700 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636704 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636706 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636709 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636711 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636714 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.636716 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636778 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636784 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636791 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636795 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636799 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636802 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 12 19:21:52.001755 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636806 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636810 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636815 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636817 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636821 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636824 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636827 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636830 2565 flags.go:64] FLAG: --cgroup-root="" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636833 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636836 2565 flags.go:64] FLAG: --client-ca-file="" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636839 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636842 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636846 2565 flags.go:64] FLAG: --cluster-domain="" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636849 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636853 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636856 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636859 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636863 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636866 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636869 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636872 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636875 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636878 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636880 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 12 19:21:52.002768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636883 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636886 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636890 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636893 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636896 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636898 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636901 2565 flags.go:64] FLAG: --enable-server="true" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636905 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636909 2565 flags.go:64] FLAG: --event-burst="100" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636912 2565 flags.go:64] FLAG: --event-qps="50" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636916 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636919 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636922 2565 flags.go:64] FLAG: --eviction-hard="" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636926 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636929 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636931 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636934 2565 flags.go:64] FLAG: --eviction-soft="" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636937 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636940 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636943 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636945 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636948 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636952 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636955 2565 flags.go:64] FLAG: --feature-gates="" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636958 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 12 19:21:52.003872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636961 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636964 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636967 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636970 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636973 2565 flags.go:64] FLAG: --help="false" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636976 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636979 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636982 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636984 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636987 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636990 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636993 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636996 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.636999 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637002 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637005 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637008 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637012 2565 flags.go:64] FLAG: --kube-reserved="" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637015 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637017 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637021 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637023 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637026 2565 flags.go:64] FLAG: --lock-file="" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637029 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 12 19:21:52.004966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637031 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637034 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637039 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637042 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637045 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637049 2565 flags.go:64] FLAG: --logging-format="text" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637052 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637055 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637057 2565 flags.go:64] FLAG: --manifest-url="" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637060 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637064 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637067 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637071 2565 flags.go:64] FLAG: --max-pods="110" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637073 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637076 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637079 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637082 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637085 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637088 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637091 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637098 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637101 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637104 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 12 19:21:52.006059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637107 2565 flags.go:64] FLAG: --pod-cidr="" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637110 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637117 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637120 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637123 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637126 2565 flags.go:64] FLAG: --port="10250" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637129 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637131 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-093f1b736f319598e" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637134 2565 flags.go:64] FLAG: --qos-reserved="" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637137 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637140 2565 flags.go:64] FLAG: --register-node="true" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637143 2565 flags.go:64] FLAG: --register-with-taints="" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637146 2565 flags.go:64] FLAG: --registry-burst="10" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637149 2565 flags.go:64] FLAG: --registry-qps="5" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637152 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637156 2565 flags.go:64] FLAG: --reserved-memory="" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637160 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637163 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637166 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637168 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637171 2565 flags.go:64] FLAG: --runonce="false" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637187 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637190 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637193 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637196 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 12 19:21:52.009436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637199 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637202 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637205 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637207 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637210 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637213 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637216 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637219 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637222 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637228 2565 flags.go:64] FLAG: --system-cgroups="" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637231 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637236 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637239 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637242 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637246 2565 flags.go:64] FLAG: --tls-min-version="" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637248 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637251 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637254 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637257 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637260 2565 flags.go:64] FLAG: --v="2" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637263 2565 flags.go:64] FLAG: --version="false" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637267 2565 flags.go:64] FLAG: --vmodule="" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637272 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.637275 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637381 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:21:52.010508 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637385 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637388 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637391 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637394 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637397 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637401 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637405 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637409 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637413 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637416 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637419 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637422 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637425 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637427 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637431 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637433 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637438 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637441 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637443 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:21:52.015607 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637446 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637448 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637451 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637453 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637456 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637459 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637462 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637464 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637466 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637469 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637474 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637476 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637479 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637481 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637484 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637487 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637489 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637492 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637494 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637496 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:21:52.016192 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637499 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637502 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637504 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637507 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637509 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637512 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637515 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637517 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637519 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637524 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637526 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637529 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637532 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637534 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637536 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637539 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637541 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637544 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637546 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637549 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:21:52.016749 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637551 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637554 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637557 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637560 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637562 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637565 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637567 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637570 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637573 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637575 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637578 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637580 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637583 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637585 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637588 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637590 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637592 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637595 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637598 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:21:52.017325 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637600 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637603 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637606 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637609 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637612 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637614 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637617 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637620 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637622 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637625 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637627 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637629 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637632 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637636 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637639 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637643 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637646 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637649 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637651 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:21:52.204168 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.637654 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.638474 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.645995 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.646008 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.646026 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.646031 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646094 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646099 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646103 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646106 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646109 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646112 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646114 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646118 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646120 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:21:52.204974 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646123 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646126 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646128 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646137 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646141 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646143 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646145 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646148 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646150 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646153 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646156 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646158 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646161 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646163 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646167 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646169 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646172 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646191 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646194 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:21:52.205593 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646197 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646200 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646203 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646205 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646208 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646210 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646213 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646216 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646218 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646221 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646225 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646229 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646232 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646234 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646237 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646240 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646242 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646251 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646254 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:21:52.206415 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646256 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646259 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646262 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646267 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646270 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646272 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646275 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646278 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646280 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646283 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646285 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646289 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646292 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646295 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646299 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646302 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646304 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646307 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646310 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646312 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:21:52.207025 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646315 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646317 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646320 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646322 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646325 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646327 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646330 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646332 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646334 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646337 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646339 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646349 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646353 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646355 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646358 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646364 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646367 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646369 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646372 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646374 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:21:52.207846 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646377 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646380 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646382 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646385 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646389 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646392 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646394 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646397 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646399 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646402 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646404 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646407 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.646411 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646585 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646590 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 12 19:21:52.208423 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646593 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646596 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646599 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646602 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646605 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646607 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646610 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646613 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646615 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646623 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646626 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646628 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646631 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646635 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646637 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646640 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646649 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646652 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646654 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646657 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 12 19:21:52.208854 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646659 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646663 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646665 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646668 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646670 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646673 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646675 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646679 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646682 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646685 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646688 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646691 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646694 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646696 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646699 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646702 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646704 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646707 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646709 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 12 19:21:52.209397 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646712 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646714 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646717 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646725 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646727 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646730 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646733 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646736 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646739 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646742 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646744 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646746 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646749 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646753 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646757 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646760 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646763 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646766 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646769 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 12 19:21:52.209927 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646771 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646774 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646777 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646780 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646782 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646785 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646788 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646791 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646794 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646796 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646799 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646802 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646805 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646807 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646810 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646812 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646814 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646823 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646826 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646830 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 12 19:21:52.210431 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646832 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646835 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646838 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646841 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646843 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646846 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646849 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646854 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646857 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646859 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646862 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646864 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646867 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646869 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646872 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646874 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646877 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646879 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 12 19:21:52.210985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:51.646882 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.646886 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.647007 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.649417 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.651118 2565 server.go:1019] "Starting client certificate rotation" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.651210 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.652108 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.657430 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.660665 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.669964 2565 log.go:25] "Validated CRI v1 runtime API" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.675414 2565 log.go:25] "Validated CRI v1 image API" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.677100 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 12 19:21:52.211545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.679559 2565 fs.go:135] Filesystem UUIDs: map[34b55265-2eb5-48b9-8f3b-780278383456:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a6499feb-ed11-4f80-8213-df99dd64f912:/dev/nvme0n1p4] Jul 12 19:21:52.211919 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.679574 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 12 19:21:52.211989 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.686262 2565 manager.go:216] Machine: {Timestamp:2026-07-12 19:21:51.684676991 +0000 UTC m=+0.367155442 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100798 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b9aa24dfb52d12e45d52168bd5f3f SystemUUID:ec2b9aa2-4dfb-52d1-2e45-d52168bd5f3f BootID:7b039ae4-9ed7-4db8-84f5-5a07a1fbe086 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:22:37:71:41:73 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:22:37:71:41:73 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:5c:06:ea:07:05 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 12 19:21:52.211989 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.686361 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 12 19:21:52.211989 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.686436 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 12 19:21:52.211989 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.688471 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:21:52.211989 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.689961 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 12 19:21:52.211989 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.689988 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-249.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 12 19:21:52.211989 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.691355 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.691375 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.691382 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.691393 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.693322 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698127 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698254 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698283 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698294 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698309 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698320 2565 kubelet.go:398] "Adding apiserver pod source" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698328 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698688 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.698703 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.700866 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.706966 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707011 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707289 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707305 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707312 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707317 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707327 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707336 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707893 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707905 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707916 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707923 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707936 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.707945 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.709855 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.709888 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.710094 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.710095 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.712631 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-249.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.713072 2565 server.go:1268] "Started kubelet" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.713146 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.713197 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.713230 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.714832 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 12 19:21:52.212288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.716027 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.717219 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.720158 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-249.ec2.internal.18c1a125ef1e9107 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-249.ec2.internal,UID:ip-10-0-141-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-249.ec2.internal,},FirstTimestamp:2026-07-12 19:21:51.713046791 +0000 UTC m=+0.395525242,LastTimestamp:2026-07-12 19:21:51.713046791 +0000 UTC m=+0.395525242,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-249.ec2.internal,}" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.724483 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.724489 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.724558 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725204 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725212 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725219 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725361 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725371 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725493 2565 factory.go:153] Registering CRI-O factory Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725506 2565 factory.go:223] Registration of the crio container factory successfully Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725551 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725558 2565 factory.go:55] Registering systemd factory Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725565 2565 factory.go:223] Registration of the systemd container factory successfully Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725588 2565 factory.go:103] Registering Raw factory Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725597 2565 manager.go:1196] Started watching for new ooms in manager Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.725980 2565 manager.go:349] Starting recovery of all containers Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.727898 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.727992 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-249.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.731931 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fjh8w" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.735649 2565 manager.go:354] Recovery completed Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.738582 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fjh8w" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.741057 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.742725 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.742747 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.742758 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.743233 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.743245 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.743261 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.743247 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:21:52.213639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.743944 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.743969 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.743979 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.744377 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-249.ec2.internal.18c1a125f0e390d8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-249.ec2.internal,UID:ip-10-0-141-249.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-249.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-249.ec2.internal,},FirstTimestamp:2026-07-12 19:21:51.742734552 +0000 UTC m=+0.425213004,LastTimestamp:2026-07-12 19:21:51.742734552 +0000 UTC m=+0.425213004,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-249.ec2.internal,}" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.745237 2565 policy_none.go:49] "None policy: Start" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.745249 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.745259 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.747193 2565 policy_none.go:47] "Start" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.824617 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.827748 2565 manager.go:337] "Starting Device Plugin manager" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.827777 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.827787 2565 server.go:87] "Starting device plugin registration server" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.828019 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.828028 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.828149 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.828255 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.828264 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.828682 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.828742 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.854006 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.855143 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.855202 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.855234 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.855271 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.857513 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.928850 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.929691 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.929720 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.929743 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.929769 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.938170 2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-249.ec2.internal\" not found" node="ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.941610 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.214515 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.941627 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-249.ec2.internal\": node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.955881 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal"] Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.955933 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.957209 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.957234 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.957246 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.959337 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.959447 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.959484 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.959964 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.959989 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.959998 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.959964 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.960065 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.960085 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.962678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.962715 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.963301 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientMemory" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.963327 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasNoDiskPressure" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:51.963351 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeHasSufficientPID" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.988811 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-249.ec2.internal\" not found" node="ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.994494 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-249.ec2.internal\" not found" node="ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:51.997629 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.026926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/800c22cd55f39cf694aa67ac1f49a873-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal\" (UID: \"800c22cd55f39cf694aa67ac1f49a873\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.026950 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/800c22cd55f39cf694aa67ac1f49a873-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal\" (UID: \"800c22cd55f39cf694aa67ac1f49a873\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.026968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c0de6f6d239334ca28d971fb295f674-config\") pod \"kube-apiserver-proxy-ip-10-0-141-249.ec2.internal\" (UID: \"8c0de6f6d239334ca28d971fb295f674\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.310151 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:52.098386 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:52.311217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.127750 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/800c22cd55f39cf694aa67ac1f49a873-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal\" (UID: \"800c22cd55f39cf694aa67ac1f49a873\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.311217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.127810 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c0de6f6d239334ca28d971fb295f674-config\") pod \"kube-apiserver-proxy-ip-10-0-141-249.ec2.internal\" (UID: \"8c0de6f6d239334ca28d971fb295f674\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.311217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.127827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/800c22cd55f39cf694aa67ac1f49a873-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal\" (UID: \"800c22cd55f39cf694aa67ac1f49a873\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.311217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.127854 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/800c22cd55f39cf694aa67ac1f49a873-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal\" (UID: \"800c22cd55f39cf694aa67ac1f49a873\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.311217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.127891 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/800c22cd55f39cf694aa67ac1f49a873-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal\" (UID: \"800c22cd55f39cf694aa67ac1f49a873\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.311217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:52.127908 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c0de6f6d239334ca28d971fb295f674-config\") pod \"kube-apiserver-proxy-ip-10-0-141-249.ec2.internal\" (UID: \"8c0de6f6d239334ca28d971fb295f674\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" Jul 12 19:21:52.311217 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:52.198578 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:53.536805 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:53.536858 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:53.536911 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-11 19:16:52 +0000 UTC" deadline="2027-12-16 06:53:01.151878054 +0000 UTC" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:53.536935 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:53.536942 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12515h31m7.61493976s" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:53.536865 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:53.537091 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:53.537146 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: Get \"https://a6be1fc95a3f745119cd290ba6aeb844-0ff466a8267f8eb1.elb.us-east-1.amazonaws.com:6443/api/v1/nodes?fieldSelector=metadata.name%3Dip-10-0-141-249.ec2.internal&limit=500&resourceVersion=0\": http2: client connection force closed via ClientConn.Close" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 12 19:21:53.537202 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:53.537166 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: Get \"https://a6be1fc95a3f745119cd290ba6aeb844-0ff466a8267f8eb1.elb.us-east-1.amazonaws.com:6443/api/v1/services?fieldSelector=spec.clusterIP%21%3DNone&limit=500&resourceVersion=0\": http2: client connection force closed via ClientConn.Close" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 12 19:21:53.697956 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:53.548855 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:21:53.697956 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:53.637254 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:53.810737 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:53.738067 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:53.914561 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:53.838741 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:53.939314 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:53.939292 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.039810 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.039742 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.140536 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.140503 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.227061 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:54.227015 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c0de6f6d239334ca28d971fb295f674.slice/crio-9cb7f7974e2240cd3c54819873324cf5e1555326202e241b6819b929f96f476d WatchSource:0}: Error finding container 9cb7f7974e2240cd3c54819873324cf5e1555326202e241b6819b929f96f476d: Status 404 returned error can't find the container with id 9cb7f7974e2240cd3c54819873324cf5e1555326202e241b6819b929f96f476d Jul 12 19:21:54.227474 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:54.227454 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800c22cd55f39cf694aa67ac1f49a873.slice/crio-accee22fdcdf0754ae9137938489ff58d475b01aeecc6dc03621c3553b6c6e58 WatchSource:0}: Error finding container accee22fdcdf0754ae9137938489ff58d475b01aeecc6dc03621c3553b6c6e58: Status 404 returned error can't find the container with id accee22fdcdf0754ae9137938489ff58d475b01aeecc6dc03621c3553b6c6e58 Jul 12 19:21:54.233581 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:54.233563 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:21:54.240597 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.240578 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.340897 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.340804 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.441430 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.441394 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.540441 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:54.540384 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" event={"ID":"800c22cd55f39cf694aa67ac1f49a873","Type":"ContainerStarted","Data":"accee22fdcdf0754ae9137938489ff58d475b01aeecc6dc03621c3553b6c6e58"} Jul 12 19:21:54.541465 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.541446 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.541564 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:54.541549 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" event={"ID":"8c0de6f6d239334ca28d971fb295f674","Type":"ContainerStarted","Data":"9cb7f7974e2240cd3c54819873324cf5e1555326202e241b6819b929f96f476d"} Jul 12 19:21:54.601496 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:54.601415 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:21:54.642125 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.642083 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.742747 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.742701 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.843494 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.843460 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:54.943638 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:54.943550 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:55.044357 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.044321 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:55.144814 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.144783 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:55.245877 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.245802 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:55.346411 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.346374 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-249.ec2.internal\" not found" Jul 12 19:21:55.361103 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.361076 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:21:55.425227 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.425176 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" Jul 12 19:21:55.438110 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.438081 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:21:55.439696 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.439670 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" Jul 12 19:21:55.447951 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.447933 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 12 19:21:55.538551 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.538455 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 12 19:21:55.538708 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.538474 2565 apiserver.go:52] "Watching apiserver" Jul 12 19:21:55.547248 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.547224 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 12 19:21:55.547674 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.547649 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85","openshift-dns/node-resolver-n6xsf","openshift-image-registry/node-ca-zgx4p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal","openshift-multus/multus-additional-cni-plugins-dwllz","openshift-network-operator/iptables-alerter-4tgzk","kube-system/konnectivity-agent-dqlj7","openshift-cluster-node-tuning-operator/tuned-qjds6","openshift-multus/multus-79cw7","openshift-multus/network-metrics-daemon-ggg8r","openshift-network-diagnostics/network-check-target-kpjl6","openshift-ovn-kubernetes/ovnkube-node-gnfrs","kube-system/global-pull-secret-syncer-bblrg"] Jul 12 19:21:55.549220 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.549201 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 12 19:21:55.550422 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.550401 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.552804 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.552777 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:55.552898 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.552863 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:21:55.553127 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.553108 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 12 19:21:55.553127 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.553116 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 12 19:21:55.553287 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.553128 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:21:55.553287 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.553174 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p98w2\"" Jul 12 19:21:55.555889 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.555846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.558169 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.558150 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 12 19:21:55.558279 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.558152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.558279 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.558214 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 12 19:21:55.558279 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.558214 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sf9cw\"" Jul 12 19:21:55.560660 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.560439 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 12 19:21:55.560660 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.560510 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 12 19:21:55.560660 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.560607 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tgsv7\"" Jul 12 19:21:55.560858 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.560716 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 12 19:21:55.560858 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.560793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.560956 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.560851 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:21:55.563100 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.563080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.565959 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.565934 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 12 19:21:55.566055 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.566001 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 12 19:21:55.566114 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.565943 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 12 19:21:55.566170 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.565943 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-b4d9h\"" Jul 12 19:21:55.567383 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.567364 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.567862 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.567825 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 12 19:21:55.568159 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.568129 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 12 19:21:55.570066 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.570045 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:21:55.570066 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.570059 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 12 19:21:55.570256 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.570210 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g58wf\"" Jul 12 19:21:55.570433 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.570417 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.573045 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.573025 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8xhgn" Jul 12 19:21:55.573501 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.573479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.573588 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.573525 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k296g\"" Jul 12 19:21:55.574029 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.573898 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 12 19:21:55.574029 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.573928 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 12 19:21:55.574177 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.574144 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 12 19:21:55.574177 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.574155 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 12 19:21:55.574177 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.574157 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 12 19:21:55.574367 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.574175 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 12 19:21:55.575872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.575845 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cwfpd\"" Jul 12 19:21:55.576171 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.576152 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 12 19:21:55.576171 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.576166 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 12 19:21:55.576499 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.576480 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 12 19:21:55.578063 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.578042 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.580539 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.580516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:55.580641 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.580540 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 12 19:21:55.580702 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.580523 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lv8p7\"" Jul 12 19:21:55.581221 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.581203 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8xhgn" Jul 12 19:21:55.582786 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.582766 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 12 19:21:55.582881 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.582771 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 12 19:21:55.582881 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.582805 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4cprc\"" Jul 12 19:21:55.583214 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.583197 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:55.583306 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.583283 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:21:55.627384 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.627362 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 12 19:21:55.647655 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647632 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-hosts-file\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.647752 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647666 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-tmp-dir\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.647752 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-lib-modules\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.647752 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-slash\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.647752 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-netns\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.647946 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-os-release\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.647946 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-device-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.647946 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9575ef7a-efbb-434d-a849-60c5d68d5f0f-konnectivity-ca\") pod \"konnectivity-agent-dqlj7\" (UID: \"9575ef7a-efbb-434d-a849-60c5d68d5f0f\") " pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:55.647946 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647869 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-var-lib-kubelet\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.647946 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647892 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-ovn\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.647946 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647912 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-daemon-config\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.647946 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43101f96-47a6-4a66-b3bc-36244a83785b-serviceca\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647961 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.647983 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxqgz\" (UniqueName: \"kubernetes.io/projected/9985ac65-ec61-43f7-acac-96a8784118a1-kube-api-access-kxqgz\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl8sl\" (UniqueName: \"kubernetes.io/projected/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-kube-api-access-xl8sl\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-systemd\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648090 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-node-log\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648116 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-k8s-cni-cncf-io\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648141 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-socket-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648165 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-sys-fs\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-modprobe-d\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648230 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-host\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-cni-binary-copy\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.648275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648279 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-cni-multus\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648308 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648358 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648383 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c57f6e17-8e69-4af8-9a1e-540b29c4874c-kubelet-config\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-cni-netd\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648482 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovnkube-config\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-cni-bin\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648562 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-registration-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6425c1a7-0063-48ea-83dc-b7096424cfa1-host-slash\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648620 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-systemd-units\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.648774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648639 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-kubelet\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648654 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkcq\" (UniqueName: \"kubernetes.io/projected/6425c1a7-0063-48ea-83dc-b7096424cfa1-kube-api-access-7kkcq\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648680 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysconfig\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysctl-d\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-sys\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648783 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovn-node-metrics-cert\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-etc-kubernetes\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcs47\" (UniqueName: \"kubernetes.io/projected/43101f96-47a6-4a66-b3bc-36244a83785b-kube-api-access-zcs47\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648852 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgpw6\" (UniqueName: \"kubernetes.io/projected/9710beb1-c58c-4188-92b1-61ebc42865e2-kube-api-access-kgpw6\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648874 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c57f6e17-8e69-4af8-9a1e-540b29c4874c-dbus\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-kubernetes\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl64j\" (UniqueName: \"kubernetes.io/projected/1f9dac69-51e3-4c3f-91db-622bcd2c0790-kube-api-access-nl64j\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-etc-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.649434 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-log-socket\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.648986 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-conf-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43101f96-47a6-4a66-b3bc-36244a83785b-host\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649035 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-systemd\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649106 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-multus-certs\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649145 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysctl-conf\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-kubelet\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649263 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-socket-dir-parent\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649296 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-hostroot\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649322 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkv9\" (UniqueName: \"kubernetes.io/projected/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-kube-api-access-8vkv9\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649363 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9575ef7a-efbb-434d-a849-60c5d68d5f0f-agent-certs\") pod \"konnectivity-agent-dqlj7\" (UID: \"9575ef7a-efbb-434d-a849-60c5d68d5f0f\") " pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9dac69-51e3-4c3f-91db-622bcd2c0790-tmp\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-run-netns\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-cni-bin\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-env-overrides\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649495 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovnkube-script-lib\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-cni-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-cnibin\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-run\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-tuned\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649642 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-os-release\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649684 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-cnibin\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6425c1a7-0063-48ea-83dc-b7096424cfa1-iptables-alerter-script\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649747 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-var-lib-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.650760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-system-cni-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.651180 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-system-cni-dir\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.651180 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.651180 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649846 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rk8\" (UniqueName: \"kubernetes.io/projected/a8899441-0b5a-4714-9743-f03b4cc1ed86-kube-api-access-26rk8\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:55.651180 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.649872 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mssqp\" (UniqueName: \"kubernetes.io/projected/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-kube-api-access-mssqp\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.750541 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43101f96-47a6-4a66-b3bc-36244a83785b-host\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-systemd\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750582 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43101f96-47a6-4a66-b3bc-36244a83785b-host\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-multus-certs\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750638 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-systemd\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750655 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysctl-conf\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.750724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750718 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-multus-certs\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750720 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-kubelet\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.750746 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750753 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-kubelet\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750795 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-socket-dir-parent\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.750861 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs podName:a8899441-0b5a-4714-9743-f03b4cc1ed86 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:56.250816823 +0000 UTC m=+4.933295283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs") pod "network-metrics-daemon-ggg8r" (UID: "a8899441-0b5a-4714-9743-f03b4cc1ed86") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750881 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-hostroot\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-run-ovn-kubernetes\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkv9\" (UniqueName: \"kubernetes.io/projected/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-kube-api-access-8vkv9\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-hostroot\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysctl-conf\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9575ef7a-efbb-434d-a849-60c5d68d5f0f-agent-certs\") pod \"konnectivity-agent-dqlj7\" (UID: \"9575ef7a-efbb-434d-a849-60c5d68d5f0f\") " pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750959 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-socket-dir-parent\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751268 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.750986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9dac69-51e3-4c3f-91db-622bcd2c0790-tmp\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-run-netns\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751043 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-cni-bin\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-env-overrides\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751096 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-run-netns\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-cni-bin\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751184 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovnkube-script-lib\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-cni-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-cnibin\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751320 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-cnibin\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-run\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751352 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-tuned\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751409 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-cni-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751417 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-os-release\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.751968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-cnibin\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-os-release\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751480 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6425c1a7-0063-48ea-83dc-b7096424cfa1-iptables-alerter-script\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751505 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-var-lib-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751521 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-cnibin\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-system-cni-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751601 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-system-cni-dir\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751639 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-env-overrides\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751654 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rk8\" (UniqueName: \"kubernetes.io/projected/a8899441-0b5a-4714-9743-f03b4cc1ed86-kube-api-access-26rk8\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751685 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mssqp\" (UniqueName: \"kubernetes.io/projected/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-kube-api-access-mssqp\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751728 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-hosts-file\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-tmp-dir\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751760 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-system-cni-dir\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.752738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751777 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-run\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751783 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-lib-modules\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751809 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-slash\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751834 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-netns\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-system-cni-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751862 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-os-release\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-device-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9575ef7a-efbb-434d-a849-60c5d68d5f0f-konnectivity-ca\") pod \"konnectivity-agent-dqlj7\" (UID: \"9575ef7a-efbb-434d-a849-60c5d68d5f0f\") " pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-var-lib-kubelet\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751949 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-lib-modules\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-ovn\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-daemon-config\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-hosts-file\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43101f96-47a6-4a66-b3bc-36244a83785b-serviceca\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752012 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.753483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6425c1a7-0063-48ea-83dc-b7096424cfa1-iptables-alerter-script\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752076 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-device-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxqgz\" (UniqueName: \"kubernetes.io/projected/9985ac65-ec61-43f7-acac-96a8784118a1-kube-api-access-kxqgz\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl8sl\" (UniqueName: \"kubernetes.io/projected/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-kube-api-access-xl8sl\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752135 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-slash\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752165 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-systemd\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752190 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-node-log\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752204 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9710beb1-c58c-4188-92b1-61ebc42865e2-os-release\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752215 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-k8s-cni-cncf-io\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752241 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-etc-selinux\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-socket-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-sys-fs\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-modprobe-d\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.754235 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752346 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-tmp-dir\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-netns\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752417 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-var-lib-kubelet\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752349 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-host\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752449 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovnkube-script-lib\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.751809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-var-lib-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-cni-binary-copy\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752496 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-node-log\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-cni-multus\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752559 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-socket-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752563 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-modprobe-d\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.754903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-sys-fs\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-cni-multus\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752641 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-run-ovn\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9575ef7a-efbb-434d-a849-60c5d68d5f0f-konnectivity-ca\") pod \"konnectivity-agent-dqlj7\" (UID: \"9575ef7a-efbb-434d-a849-60c5d68d5f0f\") " pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.752989 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-host\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753053 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-systemd\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753078 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-daemon-config\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-run-k8s-cni-cncf-io\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753156 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-cni-binary-copy\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753181 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753237 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/43101f96-47a6-4a66-b3bc-36244a83785b-serviceca\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c57f6e17-8e69-4af8-9a1e-540b29c4874c-kubelet-config\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-cni-netd\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.755587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753287 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753318 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c57f6e17-8e69-4af8-9a1e-540b29c4874c-kubelet-config\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753350 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-cni-netd\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovnkube-config\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753413 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-cni-bin\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-registration-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753469 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6425c1a7-0063-48ea-83dc-b7096424cfa1-host-slash\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753471 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753494 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-systemd-units\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-kubelet\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9985ac65-ec61-43f7-acac-96a8784118a1-registration-dir\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753548 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkcq\" (UniqueName: \"kubernetes.io/projected/6425c1a7-0063-48ea-83dc-b7096424cfa1-kube-api-access-7kkcq\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.756386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753584 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-cni-bin\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysconfig\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9710beb1-c58c-4188-92b1-61ebc42865e2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-host-var-lib-kubelet\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753629 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysctl-d\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-sys\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753708 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovn-node-metrics-cert\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-etc-kubernetes\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcs47\" (UniqueName: \"kubernetes.io/projected/43101f96-47a6-4a66-b3bc-36244a83785b-kube-api-access-zcs47\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753776 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysconfig\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgpw6\" (UniqueName: \"kubernetes.io/projected/9710beb1-c58c-4188-92b1-61ebc42865e2-kube-api-access-kgpw6\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6425c1a7-0063-48ea-83dc-b7096424cfa1-host-slash\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c57f6e17-8e69-4af8-9a1e-540b29c4874c-dbus\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-kubernetes\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753851 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-etc-kubernetes\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.757115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl64j\" (UniqueName: \"kubernetes.io/projected/1f9dac69-51e3-4c3f-91db-622bcd2c0790-kube-api-access-nl64j\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753888 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-sys\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753897 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-etc-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.753710 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-log-socket\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-conf-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753957 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovnkube-config\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.753978 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret podName:c57f6e17-8e69-4af8-9a1e-540b29c4874c nodeName:}" failed. No retries permitted until 2026-07-12 19:21:56.253954569 +0000 UTC m=+4.936433028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret") pod "global-pull-secret-syncer-bblrg" (UID: "c57f6e17-8e69-4af8-9a1e-540b29c4874c") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.754026 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-multus-conf-dir\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.754038 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c57f6e17-8e69-4af8-9a1e-540b29c4874c-dbus\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753736 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-sysctl-d\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.754081 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-kubernetes\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.754124 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-etc-openvswitch\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.754164 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-log-socket\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.753740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-systemd-units\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.757874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.755716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f9dac69-51e3-4c3f-91db-622bcd2c0790-etc-tuned\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.758657 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.756758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-ovn-node-metrics-cert\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.758657 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.756790 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9575ef7a-efbb-434d-a849-60c5d68d5f0f-agent-certs\") pod \"konnectivity-agent-dqlj7\" (UID: \"9575ef7a-efbb-434d-a849-60c5d68d5f0f\") " pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:55.758657 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.757222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f9dac69-51e3-4c3f-91db-622bcd2c0790-tmp\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.760723 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.760703 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:21:55.760723 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.760724 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:21:55.760916 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.760734 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dl2dw for pod openshift-network-diagnostics/network-check-target-kpjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:55.760916 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:55.760789 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw podName:444d7211-cd07-42fa-a693-e5a7c8d32d59 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:56.260773189 +0000 UTC m=+4.943251644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dl2dw" (UniqueName: "kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw") pod "network-check-target-kpjl6" (UID: "444d7211-cd07-42fa-a693-e5a7c8d32d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:55.763817 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.763682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rk8\" (UniqueName: \"kubernetes.io/projected/a8899441-0b5a-4714-9743-f03b4cc1ed86-kube-api-access-26rk8\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:55.763922 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.763872 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkv9\" (UniqueName: \"kubernetes.io/projected/0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7-kube-api-access-8vkv9\") pod \"multus-79cw7\" (UID: \"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7\") " pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.764917 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.764553 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcs47\" (UniqueName: \"kubernetes.io/projected/43101f96-47a6-4a66-b3bc-36244a83785b-kube-api-access-zcs47\") pod \"node-ca-zgx4p\" (UID: \"43101f96-47a6-4a66-b3bc-36244a83785b\") " pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.766813 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.765356 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkcq\" (UniqueName: \"kubernetes.io/projected/6425c1a7-0063-48ea-83dc-b7096424cfa1-kube-api-access-7kkcq\") pod \"iptables-alerter-4tgzk\" (UID: \"6425c1a7-0063-48ea-83dc-b7096424cfa1\") " pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.766813 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.765633 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgpw6\" (UniqueName: \"kubernetes.io/projected/9710beb1-c58c-4188-92b1-61ebc42865e2-kube-api-access-kgpw6\") pod \"multus-additional-cni-plugins-dwllz\" (UID: \"9710beb1-c58c-4188-92b1-61ebc42865e2\") " pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.766813 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.766402 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mssqp\" (UniqueName: \"kubernetes.io/projected/1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c-kube-api-access-mssqp\") pod \"ovnkube-node-gnfrs\" (UID: \"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c\") " pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.768103 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.768019 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl64j\" (UniqueName: \"kubernetes.io/projected/1f9dac69-51e3-4c3f-91db-622bcd2c0790-kube-api-access-nl64j\") pod \"tuned-qjds6\" (UID: \"1f9dac69-51e3-4c3f-91db-622bcd2c0790\") " pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.769519 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.769451 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxqgz\" (UniqueName: \"kubernetes.io/projected/9985ac65-ec61-43f7-acac-96a8784118a1-kube-api-access-kxqgz\") pod \"aws-ebs-csi-driver-node-xmb85\" (UID: \"9985ac65-ec61-43f7-acac-96a8784118a1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.769839 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.769813 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl8sl\" (UniqueName: \"kubernetes.io/projected/320c816e-c163-4ccb-a5fd-a6e8d2eb88df-kube-api-access-xl8sl\") pod \"node-resolver-n6xsf\" (UID: \"320c816e-c163-4ccb-a5fd-a6e8d2eb88df\") " pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.860428 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.860341 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4tgzk" Jul 12 19:21:55.870878 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.870853 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n6xsf" Jul 12 19:21:55.880848 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.880828 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zgx4p" Jul 12 19:21:55.886678 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.886656 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dwllz" Jul 12 19:21:55.893518 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.893500 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qjds6" Jul 12 19:21:55.901567 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.901543 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:21:55.908539 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.908520 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" Jul 12 19:21:55.915406 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.915387 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-79cw7" Jul 12 19:21:55.921237 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:55.921220 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:21:56.257777 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.257687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:56.257777 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.257741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:56.258018 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.257861 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:56.258018 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.257866 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:56.258018 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.257932 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret podName:c57f6e17-8e69-4af8-9a1e-540b29c4874c nodeName:}" failed. No retries permitted until 2026-07-12 19:21:57.257914228 +0000 UTC m=+5.940392693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret") pod "global-pull-secret-syncer-bblrg" (UID: "c57f6e17-8e69-4af8-9a1e-540b29c4874c") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:56.258018 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.257948 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs podName:a8899441-0b5a-4714-9743-f03b4cc1ed86 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:57.257941028 +0000 UTC m=+5.940419468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs") pod "network-metrics-daemon-ggg8r" (UID: "a8899441-0b5a-4714-9743-f03b4cc1ed86") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:56.321411 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.321381 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9575ef7a_efbb_434d_a849_60c5d68d5f0f.slice/crio-e50bce4e1449940717407be03844f1a14a059477b68faeb667e39df80e1fc897 WatchSource:0}: Error finding container e50bce4e1449940717407be03844f1a14a059477b68faeb667e39df80e1fc897: Status 404 returned error can't find the container with id e50bce4e1449940717407be03844f1a14a059477b68faeb667e39df80e1fc897 Jul 12 19:21:56.322432 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.322403 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aeac523_4ba5_4c78_bcdf_dc9d5fb8459c.slice/crio-6e78d32e1da3cea3af6277f5ae15992e086bb3e09dd6b210c02d2e652d1bac01 WatchSource:0}: Error finding container 6e78d32e1da3cea3af6277f5ae15992e086bb3e09dd6b210c02d2e652d1bac01: Status 404 returned error can't find the container with id 6e78d32e1da3cea3af6277f5ae15992e086bb3e09dd6b210c02d2e652d1bac01 Jul 12 19:21:56.323337 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.323312 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320c816e_c163_4ccb_a5fd_a6e8d2eb88df.slice/crio-3775efead674039e9b1e0099e6f76f945392b77ca6fd3a20ecbcf48a0a8fda4a WatchSource:0}: Error finding container 3775efead674039e9b1e0099e6f76f945392b77ca6fd3a20ecbcf48a0a8fda4a: Status 404 returned error can't find the container with id 3775efead674039e9b1e0099e6f76f945392b77ca6fd3a20ecbcf48a0a8fda4a Jul 12 19:21:56.326782 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.326708 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43101f96_47a6_4a66_b3bc_36244a83785b.slice/crio-58d84bb5ad7706ef2f331f6758e663b89beb55c81bb52963b2e39f405dd8432f WatchSource:0}: Error finding container 58d84bb5ad7706ef2f331f6758e663b89beb55c81bb52963b2e39f405dd8432f: Status 404 returned error can't find the container with id 58d84bb5ad7706ef2f331f6758e663b89beb55c81bb52963b2e39f405dd8432f Jul 12 19:21:56.349317 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.349295 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0107542d_e0e6_4cbe_bc59_0ff3ed1ac8b7.slice/crio-ef370741d4b36acdb520089e9f74bafc2e9abba856c8e510db31d5449ec55ed6 WatchSource:0}: Error finding container ef370741d4b36acdb520089e9f74bafc2e9abba856c8e510db31d5449ec55ed6: Status 404 returned error can't find the container with id ef370741d4b36acdb520089e9f74bafc2e9abba856c8e510db31d5449ec55ed6 Jul 12 19:21:56.351123 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.351084 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9710beb1_c58c_4188_92b1_61ebc42865e2.slice/crio-9296c07b19bf5f3bb6fbf3d6c56a61bf5332958dbfcd5d8659f047e2e557dfd8 WatchSource:0}: Error finding container 9296c07b19bf5f3bb6fbf3d6c56a61bf5332958dbfcd5d8659f047e2e557dfd8: Status 404 returned error can't find the container with id 9296c07b19bf5f3bb6fbf3d6c56a61bf5332958dbfcd5d8659f047e2e557dfd8 Jul 12 19:21:56.352339 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.352317 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6425c1a7_0063_48ea_83dc_b7096424cfa1.slice/crio-ecb8d59a8bc2714cc07644e5bd462bf5dbeb72b292a0f934017e3d7b2e521177 WatchSource:0}: Error finding container ecb8d59a8bc2714cc07644e5bd462bf5dbeb72b292a0f934017e3d7b2e521177: Status 404 returned error can't find the container with id ecb8d59a8bc2714cc07644e5bd462bf5dbeb72b292a0f934017e3d7b2e521177 Jul 12 19:21:56.353549 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.353529 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9985ac65_ec61_43f7_acac_96a8784118a1.slice/crio-3e233ac7faaa160bda0e1fee885c9679882e57c3c34182c7c5648108a6441846 WatchSource:0}: Error finding container 3e233ac7faaa160bda0e1fee885c9679882e57c3c34182c7c5648108a6441846: Status 404 returned error can't find the container with id 3e233ac7faaa160bda0e1fee885c9679882e57c3c34182c7c5648108a6441846 Jul 12 19:21:56.354357 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:21:56.354330 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f9dac69_51e3_4c3f_91db_622bcd2c0790.slice/crio-b4384bf3d220c2cca06d33af81ec99c413d72da0471bf9eea138aeacb01a7178 WatchSource:0}: Error finding container b4384bf3d220c2cca06d33af81ec99c413d72da0471bf9eea138aeacb01a7178: Status 404 returned error can't find the container with id b4384bf3d220c2cca06d33af81ec99c413d72da0471bf9eea138aeacb01a7178 Jul 12 19:21:56.357975 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.357953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:56.358188 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.358172 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:21:56.358915 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.358195 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:21:56.358915 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.358208 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dl2dw for pod openshift-network-diagnostics/network-check-target-kpjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:56.358915 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.358281 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw podName:444d7211-cd07-42fa-a693-e5a7c8d32d59 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:57.358242062 +0000 UTC m=+6.040720519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dl2dw" (UniqueName: "kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw") pod "network-check-target-kpjl6" (UID: "444d7211-cd07-42fa-a693-e5a7c8d32d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:56.423882 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.423861 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 12 19:21:56.546698 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.546447 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" event={"ID":"8c0de6f6d239334ca28d971fb295f674","Type":"ContainerStarted","Data":"5869c5978519420daf07b6f7b16b64ed96c419fff0feec469942c0a9768b29c3"} Jul 12 19:21:56.547672 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.547640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjds6" event={"ID":"1f9dac69-51e3-4c3f-91db-622bcd2c0790","Type":"ContainerStarted","Data":"b4384bf3d220c2cca06d33af81ec99c413d72da0471bf9eea138aeacb01a7178"} Jul 12 19:21:56.548720 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.548698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" event={"ID":"9985ac65-ec61-43f7-acac-96a8784118a1","Type":"ContainerStarted","Data":"3e233ac7faaa160bda0e1fee885c9679882e57c3c34182c7c5648108a6441846"} Jul 12 19:21:56.549947 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.549919 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4tgzk" event={"ID":"6425c1a7-0063-48ea-83dc-b7096424cfa1","Type":"ContainerStarted","Data":"ecb8d59a8bc2714cc07644e5bd462bf5dbeb72b292a0f934017e3d7b2e521177"} Jul 12 19:21:56.551080 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.551058 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"6e78d32e1da3cea3af6277f5ae15992e086bb3e09dd6b210c02d2e652d1bac01"} Jul 12 19:21:56.552197 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.552163 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerStarted","Data":"9296c07b19bf5f3bb6fbf3d6c56a61bf5332958dbfcd5d8659f047e2e557dfd8"} Jul 12 19:21:56.553343 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.553319 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-79cw7" event={"ID":"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7","Type":"ContainerStarted","Data":"ef370741d4b36acdb520089e9f74bafc2e9abba856c8e510db31d5449ec55ed6"} Jul 12 19:21:56.554380 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.554360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zgx4p" event={"ID":"43101f96-47a6-4a66-b3bc-36244a83785b","Type":"ContainerStarted","Data":"58d84bb5ad7706ef2f331f6758e663b89beb55c81bb52963b2e39f405dd8432f"} Jul 12 19:21:56.555402 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.555376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n6xsf" event={"ID":"320c816e-c163-4ccb-a5fd-a6e8d2eb88df","Type":"ContainerStarted","Data":"3775efead674039e9b1e0099e6f76f945392b77ca6fd3a20ecbcf48a0a8fda4a"} Jul 12 19:21:56.556497 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.556479 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dqlj7" event={"ID":"9575ef7a-efbb-434d-a849-60c5d68d5f0f","Type":"ContainerStarted","Data":"e50bce4e1449940717407be03844f1a14a059477b68faeb667e39df80e1fc897"} Jul 12 19:21:56.561936 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.561897 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-249.ec2.internal" podStartSLOduration=1.5618844840000001 podStartE2EDuration="1.561884484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:21:56.56132366 +0000 UTC m=+5.243802127" watchObservedRunningTime="2026-07-12 19:21:56.561884484 +0000 UTC m=+5.244362947" Jul 12 19:21:56.585121 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.585073 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:16:55 +0000 UTC" deadline="2028-01-26 21:21:14.240922378 +0000 UTC" Jul 12 19:21:56.585121 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.585116 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13513h59m17.655808889s" Jul 12 19:21:56.855759 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.855637 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:56.855759 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:56.855690 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:56.855994 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.855780 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:21:56.855994 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:56.855931 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:21:57.265692 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.265602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:57.265692 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.265666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:57.265941 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.265833 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:57.265941 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.265897 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs podName:a8899441-0b5a-4714-9743-f03b4cc1ed86 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:59.265879937 +0000 UTC m=+7.948358393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs") pod "network-metrics-daemon-ggg8r" (UID: "a8899441-0b5a-4714-9743-f03b4cc1ed86") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:57.266401 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.266380 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:57.266504 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.266443 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret podName:c57f6e17-8e69-4af8-9a1e-540b29c4874c nodeName:}" failed. No retries permitted until 2026-07-12 19:21:59.266426642 +0000 UTC m=+7.948905098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret") pod "global-pull-secret-syncer-bblrg" (UID: "c57f6e17-8e69-4af8-9a1e-540b29c4874c") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:57.367125 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.367085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:57.367541 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.367519 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:21:57.367651 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.367547 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:21:57.367651 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.367562 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dl2dw for pod openshift-network-diagnostics/network-check-target-kpjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:57.367651 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.367626 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw podName:444d7211-cd07-42fa-a693-e5a7c8d32d59 nodeName:}" failed. No retries permitted until 2026-07-12 19:21:59.367605974 +0000 UTC m=+8.050084444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dl2dw" (UniqueName: "kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw") pod "network-check-target-kpjl6" (UID: "444d7211-cd07-42fa-a693-e5a7c8d32d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:57.586004 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.585902 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-11 19:16:55 +0000 UTC" deadline="2028-01-20 23:54:58.409652169 +0000 UTC" Jul 12 19:21:57.586004 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.585953 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13372h33m0.82370424s" Jul 12 19:21:57.590381 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.588974 2565 generic.go:356] "Generic (PLEG): container finished" podID="800c22cd55f39cf694aa67ac1f49a873" containerID="5a4311877e3f4ac748b9a91af1f5d476cccb92f8ae325053c1fe0359eed38716" exitCode=0 Jul 12 19:21:57.590381 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.590149 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" event={"ID":"800c22cd55f39cf694aa67ac1f49a873","Type":"ContainerDied","Data":"5a4311877e3f4ac748b9a91af1f5d476cccb92f8ae325053c1fe0359eed38716"} Jul 12 19:21:57.859081 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:57.858431 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:57.859081 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:57.858581 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:21:58.618312 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:58.618100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" event={"ID":"800c22cd55f39cf694aa67ac1f49a873","Type":"ContainerStarted","Data":"6cec71d9146cb81ad633838ae352a11b2e34e698c11181571488a40112bc3417"} Jul 12 19:21:58.856212 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:58.855471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:58.856212 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:58.855609 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:21:58.856212 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:58.855719 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:58.856212 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:58.855809 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:21:59.286478 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:59.286438 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:21:59.286660 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:59.286495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:59.286660 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.286635 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:59.286787 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.286699 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs podName:a8899441-0b5a-4714-9743-f03b4cc1ed86 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:03.286679311 +0000 UTC m=+11.969157765 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs") pod "network-metrics-daemon-ggg8r" (UID: "a8899441-0b5a-4714-9743-f03b4cc1ed86") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:21:59.287149 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.287129 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:59.287351 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.287188 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret podName:c57f6e17-8e69-4af8-9a1e-540b29c4874c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:03.28717294 +0000 UTC m=+11.969651395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret") pod "global-pull-secret-syncer-bblrg" (UID: "c57f6e17-8e69-4af8-9a1e-540b29c4874c") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:21:59.387620 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:59.387575 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:21:59.388231 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.387794 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:21:59.388231 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.387822 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:21:59.388231 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.387836 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dl2dw for pod openshift-network-diagnostics/network-check-target-kpjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:59.388231 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.387903 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw podName:444d7211-cd07-42fa-a693-e5a7c8d32d59 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:03.387884293 +0000 UTC m=+12.070362734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dl2dw" (UniqueName: "kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw") pod "network-check-target-kpjl6" (UID: "444d7211-cd07-42fa-a693-e5a7c8d32d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:21:59.856334 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:21:59.855842 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:21:59.856334 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:21:59.855982 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:00.856099 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:00.856056 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:00.856303 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:00.856203 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:00.856627 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:00.856606 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:00.857006 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:00.856710 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:01.856398 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:01.856362 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:01.856631 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:01.856489 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:02.855706 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:02.855678 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:02.855881 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:02.855785 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:02.856104 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:02.856080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:02.856260 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:02.856211 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:03.323008 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:03.322574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:03.323008 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:03.322628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:03.323008 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.322724 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:03.323008 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.322788 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs podName:a8899441-0b5a-4714-9743-f03b4cc1ed86 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:11.322768855 +0000 UTC m=+20.005247307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs") pod "network-metrics-daemon-ggg8r" (UID: "a8899441-0b5a-4714-9743-f03b4cc1ed86") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:03.323617 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.322724 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:03.323617 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.323158 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret podName:c57f6e17-8e69-4af8-9a1e-540b29c4874c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:11.323132484 +0000 UTC m=+20.005610927 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret") pod "global-pull-secret-syncer-bblrg" (UID: "c57f6e17-8e69-4af8-9a1e-540b29c4874c") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:03.423971 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:03.423933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:03.424138 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.424107 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:03.424138 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.424126 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:03.424138 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.424137 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dl2dw for pod openshift-network-diagnostics/network-check-target-kpjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:03.424341 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.424206 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw podName:444d7211-cd07-42fa-a693-e5a7c8d32d59 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:11.424171303 +0000 UTC m=+20.106649755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dl2dw" (UniqueName: "kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw") pod "network-check-target-kpjl6" (UID: "444d7211-cd07-42fa-a693-e5a7c8d32d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:03.855569 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:03.855538 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:03.855739 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:03.855695 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:04.855706 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:04.855674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:04.856167 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:04.855801 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:04.856167 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:04.856152 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:04.856295 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:04.856261 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:05.856220 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:05.856162 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:05.856671 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:05.856326 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:06.855632 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:06.855591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:06.855632 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:06.855621 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:06.855883 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:06.855733 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:06.855940 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:06.855878 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:07.856296 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:07.856255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:07.856761 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:07.856412 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:08.855646 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:08.855612 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:08.855832 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:08.855612 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:08.855832 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:08.855752 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:08.855921 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:08.855864 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:09.858390 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:09.858364 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:09.858837 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:09.858492 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:10.856060 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:10.856032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:10.856226 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:10.856039 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:10.856226 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:10.856123 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:10.856226 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:10.856210 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:11.385288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:11.385256 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:11.385754 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:11.385424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:11.385754 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.385429 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:11.385754 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.385478 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:11.385754 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.385506 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs podName:a8899441-0b5a-4714-9743-f03b4cc1ed86 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:27.385484508 +0000 UTC m=+36.067962966 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs") pod "network-metrics-daemon-ggg8r" (UID: "a8899441-0b5a-4714-9743-f03b4cc1ed86") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:11.385754 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.385533 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret podName:c57f6e17-8e69-4af8-9a1e-540b29c4874c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:27.385516808 +0000 UTC m=+36.067995269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret") pod "global-pull-secret-syncer-bblrg" (UID: "c57f6e17-8e69-4af8-9a1e-540b29c4874c") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:11.485815 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:11.485786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:11.485981 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.485953 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:11.485981 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.485978 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:11.486099 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.485991 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dl2dw for pod openshift-network-diagnostics/network-check-target-kpjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:11.486099 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.486041 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw podName:444d7211-cd07-42fa-a693-e5a7c8d32d59 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:27.486026784 +0000 UTC m=+36.168505222 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dl2dw" (UniqueName: "kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw") pod "network-check-target-kpjl6" (UID: "444d7211-cd07-42fa-a693-e5a7c8d32d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:11.856150 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:11.856079 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:11.856358 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:11.856167 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:12.856338 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:12.856285 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:12.856872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:12.856293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:12.856872 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:12.856452 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:12.856872 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:12.856481 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:13.856279 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:13.856243 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:13.856450 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:13.856397 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:14.856478 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:14.856442 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:14.856478 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:14.856484 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:14.856920 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:14.856557 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:14.856920 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:14.856624 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:15.653338 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.653078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerStarted","Data":"bcf99ae6a1fb9fbeff49ad0e599a666956a99ff50c8c90f39a3236ea5a84768f"} Jul 12 19:22:15.654941 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.654865 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-79cw7" event={"ID":"0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7","Type":"ContainerStarted","Data":"f5932d75fe71978a2e9193a648997ba803bb50afca5504737ed4bf52ab4bed89"} Jul 12 19:22:15.656992 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.656672 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zgx4p" event={"ID":"43101f96-47a6-4a66-b3bc-36244a83785b","Type":"ContainerStarted","Data":"4ba6a2d443017bdfdefd4f8acb9f9bb81107410726fe698fb8e81539bf017f21"} Jul 12 19:22:15.659226 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.659193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n6xsf" event={"ID":"320c816e-c163-4ccb-a5fd-a6e8d2eb88df","Type":"ContainerStarted","Data":"22811f30c8bc5c149af5dd8a823abdf1390cdc1b045fd449a0a9738ac26acfbb"} Jul 12 19:22:15.660693 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.660669 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dqlj7" event={"ID":"9575ef7a-efbb-434d-a849-60c5d68d5f0f","Type":"ContainerStarted","Data":"b644730d7bd884c8f99676eaa3b88c19a1030e728577ad26423fe4152da9d35a"} Jul 12 19:22:15.662302 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.662234 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qjds6" event={"ID":"1f9dac69-51e3-4c3f-91db-622bcd2c0790","Type":"ContainerStarted","Data":"c263ff65cf04506d54e47bd447eedc85131f58bcde124d47a009c38489c17740"} Jul 12 19:22:15.663833 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.663814 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" event={"ID":"9985ac65-ec61-43f7-acac-96a8784118a1","Type":"ContainerStarted","Data":"f93b993f5549c20557162135f97fecffdd19d10713e96be0dc158d4fee985199"} Jul 12 19:22:15.665717 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.665694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"aaa144c924477517772936758472f00d20f501c24a421f35bdd258e2965dc520"} Jul 12 19:22:15.674224 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.674156 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-249.ec2.internal" podStartSLOduration=20.674119356 podStartE2EDuration="20.674119356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:21:58.635713948 +0000 UTC m=+7.318192412" watchObservedRunningTime="2026-07-12 19:22:15.674119356 +0000 UTC m=+24.356597816" Jul 12 19:22:15.701338 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.701283 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n6xsf" podStartSLOduration=4.572406664 podStartE2EDuration="23.701265039s" totalImagesPullingTime="19.128858375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:15.686786503 +0000 UTC m=+24.369264964" watchObservedRunningTime="2026-07-12 19:22:15.701265039 +0000 UTC m=+24.383743502" Jul 12 19:22:15.701439 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.701389 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zgx4p" podStartSLOduration=4.592987786 podStartE2EDuration="23.701381696s" totalImagesPullingTime="19.10839391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:15.701022476 +0000 UTC m=+24.383500937" watchObservedRunningTime="2026-07-12 19:22:15.701381696 +0000 UTC m=+24.383860156" Jul 12 19:22:15.715957 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.715914 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qjds6" podStartSLOduration=4.616513843 podStartE2EDuration="23.715899583s" totalImagesPullingTime="19.09938574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:15.715431203 +0000 UTC m=+24.397909662" watchObservedRunningTime="2026-07-12 19:22:15.715899583 +0000 UTC m=+24.398378042" Jul 12 19:22:15.732825 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.732784 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-79cw7" podStartSLOduration=4.590616159 podStartE2EDuration="23.732769978s" totalImagesPullingTime="19.142153819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:15.732538438 +0000 UTC m=+24.415016924" watchObservedRunningTime="2026-07-12 19:22:15.732769978 +0000 UTC m=+24.415248437" Jul 12 19:22:15.753881 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.753821 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dqlj7" podStartSLOduration=12.925355503 podStartE2EDuration="23.753798637s" totalImagesPullingTime="10.828443134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:15.753782525 +0000 UTC m=+24.436260985" watchObservedRunningTime="2026-07-12 19:22:15.753798637 +0000 UTC m=+24.436277181" Jul 12 19:22:15.856157 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:15.856136 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:15.856289 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:15.856267 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:16.670528 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.670490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"40db33290cab6ca410efd7d88d1dd1042b771c2e906effb87b491dbe43873ca8"} Jul 12 19:22:16.671363 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.670577 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"930477d6c5ad7cbf7d7fcc3a9e43a515a6d69fa790943656a1e9f55799d843d8"} Jul 12 19:22:16.671363 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.670628 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"50d9318422a77bd7aed3676264c2e867289af045a099ae3e96040981422c67a2"} Jul 12 19:22:16.671363 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.670675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"a32bfcf7b795bbc55dc63bdb345b7e68173cd6ec8d3a8ce41eead6ac65b7bf4d"} Jul 12 19:22:16.671363 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.670719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"e10cad23c117e4a4a051ed114f6c2911aa2d1f29494781b9e35d8fa86a53cc23"} Jul 12 19:22:16.671920 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.671898 2565 generic.go:356] "Generic (PLEG): container finished" podID="9710beb1-c58c-4188-92b1-61ebc42865e2" containerID="bcf99ae6a1fb9fbeff49ad0e599a666956a99ff50c8c90f39a3236ea5a84768f" exitCode=0 Jul 12 19:22:16.672018 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.671999 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerDied","Data":"bcf99ae6a1fb9fbeff49ad0e599a666956a99ff50c8c90f39a3236ea5a84768f"} Jul 12 19:22:16.855665 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.855526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:16.855665 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.855531 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:16.855800 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:16.855663 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:16.855800 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:16.855688 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:16.992251 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:16.992231 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 12 19:22:17.550609 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:17.550490 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-12T19:22:16.992247404Z","UUID":"2857464a-9a04-4c31-bab3-d51aa82ff7f5","Handler":null,"Name":"","Endpoint":""} Jul 12 19:22:17.552435 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:17.552415 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 12 19:22:17.552546 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:17.552464 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 12 19:22:17.676898 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:17.676861 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" event={"ID":"9985ac65-ec61-43f7-acac-96a8784118a1","Type":"ContainerStarted","Data":"cd35dffd0a59259e320387935667d87797b5e0b694343955d66f5980fcc76b7b"} Jul 12 19:22:17.678577 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:17.678524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4tgzk" event={"ID":"6425c1a7-0063-48ea-83dc-b7096424cfa1","Type":"ContainerStarted","Data":"d82324831e15102bf71d920e985edd3515e6c871a06932fc6e5dcecbccc6ad74"} Jul 12 19:22:17.691608 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:17.691557 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4tgzk" podStartSLOduration=6.592813131 podStartE2EDuration="25.6915444s" totalImagesPullingTime="19.098731269s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:17.691248454 +0000 UTC m=+26.373726915" watchObservedRunningTime="2026-07-12 19:22:17.6915444 +0000 UTC m=+26.374022860" Jul 12 19:22:17.856660 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:17.856319 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:17.856660 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:17.856463 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:18.120007 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.119933 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:22:18.120548 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.120525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:22:18.683421 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.683341 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" event={"ID":"9985ac65-ec61-43f7-acac-96a8784118a1","Type":"ContainerStarted","Data":"fdb973e1c756ef85efc14cb5185647ff41e1d087f725a81bc95b4584a29aeb72"} Jul 12 19:22:18.686872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.686842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"1fca3fe1673fc7c35188140a1c374004b7529f2f4b8a0bc5240f2ded06f823e9"} Jul 12 19:22:18.687192 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.687134 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:22:18.687594 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.687575 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dqlj7" Jul 12 19:22:18.713277 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.713233 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xmb85" podStartSLOduration=4.975162368 podStartE2EDuration="26.713219415s" totalImagesPullingTime="21.738057047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:18.700064226 +0000 UTC m=+27.382542687" watchObservedRunningTime="2026-07-12 19:22:18.713219415 +0000 UTC m=+27.395697874" Jul 12 19:22:18.855802 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.855765 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:18.855963 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:18.855872 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:18.855963 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:18.855909 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:18.856071 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:18.856033 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:19.856489 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:19.856458 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:19.856963 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:19.856606 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:20.856688 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:20.856446 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:20.857093 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:20.856477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:20.857093 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:20.856793 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:20.857093 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:20.856851 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:21.697759 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.697551 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" event={"ID":"1aeac523-4ba5-4c78-bcdf-dc9d5fb8459c","Type":"ContainerStarted","Data":"be75e639d227527b046d006bfb3af1cd72b25f243bd4b3345f6e3a275eff1a68"} Jul 12 19:22:21.697916 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.697821 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:22:21.697916 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.697873 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:22:21.699398 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.699372 2565 generic.go:356] "Generic (PLEG): container finished" podID="9710beb1-c58c-4188-92b1-61ebc42865e2" containerID="0dbf5947bcc74caeee755897de0e7f4e2ac63995a74cfefc6a462095165d8e30" exitCode=0 Jul 12 19:22:21.699516 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.699413 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerDied","Data":"0dbf5947bcc74caeee755897de0e7f4e2ac63995a74cfefc6a462095165d8e30"} Jul 12 19:22:21.713102 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.713081 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:22:21.720703 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.720666 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" podStartSLOduration=10.519274993 podStartE2EDuration="29.72065372s" totalImagesPullingTime="19.201378727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:21.719226293 +0000 UTC m=+30.401704753" watchObservedRunningTime="2026-07-12 19:22:21.72065372 +0000 UTC m=+30.403132180" Jul 12 19:22:21.856753 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:21.856717 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:21.857607 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:21.856804 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:22.702416 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:22.702381 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:22:22.717118 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:22.717091 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:22:22.855842 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:22.855817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:22.855995 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:22.855817 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:22.855995 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:22.855918 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:22.856065 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:22.856000 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:23.051852 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.051525 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bblrg"] Jul 12 19:22:23.056842 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.056814 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kpjl6"] Jul 12 19:22:23.057559 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.057535 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ggg8r"] Jul 12 19:22:23.057674 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.057654 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:23.057782 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:23.057758 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:23.706160 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.706122 2565 generic.go:356] "Generic (PLEG): container finished" podID="9710beb1-c58c-4188-92b1-61ebc42865e2" containerID="b35dee8b11ce9ad5d4467d332897cdfc8f7050fe3e38994fea43b702134d0d62" exitCode=0 Jul 12 19:22:23.706412 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.706239 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerDied","Data":"b35dee8b11ce9ad5d4467d332897cdfc8f7050fe3e38994fea43b702134d0d62"} Jul 12 19:22:23.706412 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.706284 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:23.706412 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:23.706308 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:23.706412 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:23.706378 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:23.706563 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:23.706467 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:24.856321 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:24.856286 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:24.856581 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:24.856404 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:25.714709 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:25.714672 2565 generic.go:356] "Generic (PLEG): container finished" podID="9710beb1-c58c-4188-92b1-61ebc42865e2" containerID="e0f8bd474743bdae5e1a913a43a3d4b72aae9918a4217254e1e19d9303da3645" exitCode=0 Jul 12 19:22:25.714870 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:25.714723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerDied","Data":"e0f8bd474743bdae5e1a913a43a3d4b72aae9918a4217254e1e19d9303da3645"} Jul 12 19:22:25.855860 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:25.855684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:25.856032 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:25.855684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:25.856032 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:25.855946 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:25.856032 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:25.856005 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:26.855723 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:26.855693 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:26.856076 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:26.855804 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ggg8r" podUID="a8899441-0b5a-4714-9743-f03b4cc1ed86" Jul 12 19:22:27.410982 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:27.410947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:27.411170 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.411085 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:27.411170 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.411153 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret podName:c57f6e17-8e69-4af8-9a1e-540b29c4874c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:59.411134058 +0000 UTC m=+68.093612509 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret") pod "global-pull-secret-syncer-bblrg" (UID: "c57f6e17-8e69-4af8-9a1e-540b29c4874c") : object "kube-system"/"original-pull-secret" not registered Jul 12 19:22:27.411332 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.411172 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:27.411332 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.411240 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs podName:a8899441-0b5a-4714-9743-f03b4cc1ed86 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:59.411223622 +0000 UTC m=+68.093702063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs") pod "network-metrics-daemon-ggg8r" (UID: "a8899441-0b5a-4714-9743-f03b4cc1ed86") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 12 19:22:27.411332 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:27.411088 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:27.511748 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:27.511719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:27.511941 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.511921 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 12 19:22:27.512008 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.511952 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 12 19:22:27.512008 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.511965 2565 projected.go:196] Error preparing data for projected volume kube-api-access-dl2dw for pod openshift-network-diagnostics/network-check-target-kpjl6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:27.512103 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.512025 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw podName:444d7211-cd07-42fa-a693-e5a7c8d32d59 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:59.512005895 +0000 UTC m=+68.194484347 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dl2dw" (UniqueName: "kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw") pod "network-check-target-kpjl6" (UID: "444d7211-cd07-42fa-a693-e5a7c8d32d59") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 12 19:22:27.856382 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:27.856248 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:27.856382 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:27.856278 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:27.857334 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.856438 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kpjl6" podUID="444d7211-cd07-42fa-a693-e5a7c8d32d59" Jul 12 19:22:27.857334 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:27.856523 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bblrg" podUID="c57f6e17-8e69-4af8-9a1e-540b29c4874c" Jul 12 19:22:28.111542 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.111475 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-249.ec2.internal" event="NodeReady" Jul 12 19:22:28.111667 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.111599 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 12 19:22:28.142440 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.142408 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-567dd678fc-ppt7c"] Jul 12 19:22:28.148528 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.147374 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb"] Jul 12 19:22:28.148812 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.148773 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.152267 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.152211 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dx58x\"" Jul 12 19:22:28.152372 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.152360 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 12 19:22:28.152440 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.152423 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 12 19:22:28.152563 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.152546 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 12 19:22:28.153704 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.153521 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7"] Jul 12 19:22:28.153704 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.153640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.155935 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.155920 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.157027 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.157007 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bs8bw\"" Jul 12 19:22:28.157129 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.157034 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 12 19:22:28.157231 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.157211 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.157281 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.157235 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 12 19:22:28.157500 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.157482 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-lcxrh"] Jul 12 19:22:28.157678 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.157661 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.161986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.161017 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd"] Jul 12 19:22:28.161986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.161049 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 12 19:22:28.161986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.161062 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 12 19:22:28.161986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.161076 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.161986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.161076 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lhqxb\"" Jul 12 19:22:28.161986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.161226 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.161986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.161419 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.162654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.162095 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 12 19:22:28.164833 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.164816 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r"] Jul 12 19:22:28.164988 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.164969 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 12 19:22:28.165232 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.165215 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.165727 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.165708 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.166531 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.166510 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" Jul 12 19:22:28.166676 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.166559 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qfdkd\"" Jul 12 19:22:28.168863 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.164898 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 12 19:22:28.170869 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.170854 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.172204 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.171839 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 12 19:22:28.172204 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.171898 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-lc5q8\"" Jul 12 19:22:28.172204 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.172042 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.172978 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.172960 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-52s44"] Jul 12 19:22:28.173141 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.173121 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" Jul 12 19:22:28.178016 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.177484 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tbhnd\"" Jul 12 19:22:28.178016 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.177780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.178016 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.177855 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.183933 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.180311 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7"] Jul 12 19:22:28.183933 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.182833 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.185562 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.185543 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86d8bc5964-wgq5f"] Jul 12 19:22:28.185903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.185887 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.186719 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.186566 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 12 19:22:28.186719 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.186624 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-vjcmg\"" Jul 12 19:22:28.186914 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.186901 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 12 19:22:28.187310 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.187166 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.187310 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.187241 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.188150 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.188131 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 12 19:22:28.188534 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.188518 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 12 19:22:28.188623 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.188518 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-8l96j\"" Jul 12 19:22:28.188991 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.188847 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.189292 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.189272 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.189739 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.189708 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8"] Jul 12 19:22:28.189949 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.189783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.192621 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.191980 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 12 19:22:28.192621 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.192099 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.192621 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.192416 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 12 19:22:28.192621 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.192474 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jt2h5\"" Jul 12 19:22:28.192906 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.192878 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 12 19:22:28.193097 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.193077 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 12 19:22:28.193287 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.193245 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 12 19:22:28.193349 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.193311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.194880 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.194859 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6pvxk"] Jul 12 19:22:28.195048 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.195031 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:28.197101 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.197081 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.197328 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.197313 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 12 19:22:28.197402 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.197338 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4q58t\"" Jul 12 19:22:28.197578 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.197566 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.198483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.198465 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7"] Jul 12 19:22:28.198604 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.198503 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lmpsl"] Jul 12 19:22:28.198604 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.198591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.200822 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.200807 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 12 19:22:28.200980 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.200963 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vfn6k\"" Jul 12 19:22:28.201109 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201092 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 12 19:22:28.201249 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201231 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 12 19:22:28.201704 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201686 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-567dd678fc-ppt7c"] Jul 12 19:22:28.201795 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r"] Jul 12 19:22:28.201795 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201727 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb"] Jul 12 19:22:28.201795 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201745 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7"] Jul 12 19:22:28.201795 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201763 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8"] Jul 12 19:22:28.201795 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201787 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-lcxrh"] Jul 12 19:22:28.201795 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201794 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.202068 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201806 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86d8bc5964-wgq5f"] Jul 12 19:22:28.202068 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201824 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6pvxk"] Jul 12 19:22:28.202068 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd"] Jul 12 19:22:28.202068 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201891 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-52s44"] Jul 12 19:22:28.202068 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.201909 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmpsl"] Jul 12 19:22:28.203932 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.203912 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 12 19:22:28.204068 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.204051 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 12 19:22:28.204163 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.204130 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kfwdz\"" Jul 12 19:22:28.298635 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.298567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqlj7_9575ef7a-efbb-434d-a849-60c5d68d5f0f/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:28.315829 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.315802 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa31e23-7983-4949-b4cc-4fe996a91a92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.315977 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.315835 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-image-registry-private-configuration\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.315977 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.315861 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llq9d\" (UniqueName: \"kubernetes.io/projected/566fd99f-bbc7-4532-83ef-f8cce366f737-kube-api-access-llq9d\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.315977 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.315921 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa31e23-7983-4949-b4cc-4fe996a91a92-tmp\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.315977 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.315964 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2bd8\" (UniqueName: \"kubernetes.io/projected/6aa31e23-7983-4949-b4cc-4fe996a91a92-kube-api-access-m2bd8\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.315994 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkn7c\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-kube-api-access-lkn7c\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg7bs\" (UniqueName: \"kubernetes.io/projected/894b53d3-e0f9-4c31-b48f-8d2f80edd455-kube-api-access-rg7bs\") pod \"volume-data-source-validator-78495597f6-vd82r\" (UID: \"894b53d3-e0f9-4c31-b48f-8d2f80edd455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316064 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f6dd18-06c6-4622-890d-3865aae784de-cert\") pod \"ingress-canary-6pvxk\" (UID: \"91f6dd18-06c6-4622-890d-3865aae784de\") " pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316105 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316129 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3551a07-e183-40a5-9b98-1a6865047fb9-config\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316156 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-serving-cert\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.316201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2krng\" (UniqueName: \"kubernetes.io/projected/c3551a07-e183-40a5-9b98-1a6865047fb9-kube-api-access-2krng\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-ca-trust-extracted\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316323 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkdgx\" (UniqueName: \"kubernetes.io/projected/df8f3888-df8e-480d-ae68-f6b16126ee83-kube-api-access-dkdgx\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzczl\" (UniqueName: \"kubernetes.io/projected/928c8c62-9201-42ec-b710-560d624ec55c-kube-api-access-hzczl\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fd99f-bbc7-4532-83ef-f8cce366f737-tmp-dir\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/566fd99f-bbc7-4532-83ef-f8cce366f737-config-volume\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwzn\" (UniqueName: \"kubernetes.io/projected/980467bb-dbb4-4003-83f9-4754b8966b29-kube-api-access-tkwzn\") pod \"network-check-source-869dcfcdcd-4v8vd\" (UID: \"980467bb-dbb4-4003-83f9-4754b8966b29\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-stats-auth\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-certificates\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6g5f\" (UniqueName: \"kubernetes.io/projected/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-kube-api-access-t6g5f\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-installation-pull-secrets\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.316619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316576 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-bound-sa-token\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316617 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npc29\" (UniqueName: \"kubernetes.io/projected/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-kube-api-access-npc29\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-config\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316755 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316779 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316830 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6p4z\" (UniqueName: \"kubernetes.io/projected/91f6dd18-06c6-4622-890d-3865aae784de-kube-api-access-s6p4z\") pod \"ingress-canary-6pvxk\" (UID: \"91f6dd18-06c6-4622-890d-3865aae784de\") " pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-default-certificate\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316917 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316931 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6aa31e23-7983-4949-b4cc-4fe996a91a92-snapshots\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa31e23-7983-4949-b4cc-4fe996a91a92-serving-cert\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3551a07-e183-40a5-9b98-1a6865047fb9-serving-cert\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.316988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3551a07-e183-40a5-9b98-1a6865047fb9-trusted-ca\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.317058 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.317002 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/df8f3888-df8e-480d-ae68-f6b16126ee83-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.317583 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.317062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa31e23-7983-4949-b4cc-4fe996a91a92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.317583 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.317081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-trusted-ca\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.317583 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.317104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jtb\" (UniqueName: \"kubernetes.io/projected/d3cfe81c-2ae4-48de-9292-cac871d9da5c-kube-api-access-f6jtb\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:28.377930 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.377847 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dqlj7_9575ef7a-efbb-434d-a849-60c5d68d5f0f/konnectivity-agent/0.log" limit=1024 Jul 12 19:22:28.417787 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-default-certificate\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.417787 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.418044 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6aa31e23-7983-4949-b4cc-4fe996a91a92-snapshots\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.418044 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa31e23-7983-4949-b4cc-4fe996a91a92-serving-cert\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.418044 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417858 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3551a07-e183-40a5-9b98-1a6865047fb9-serving-cert\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.418044 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3551a07-e183-40a5-9b98-1a6865047fb9-trusted-ca\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.418044 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/df8f3888-df8e-480d-ae68-f6b16126ee83-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.418044 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa31e23-7983-4949-b4cc-4fe996a91a92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.418044 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.417960 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-trusted-ca\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.418504 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418438 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.418635 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418539 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6aa31e23-7983-4949-b4cc-4fe996a91a92-snapshots\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418719 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jtb\" (UniqueName: \"kubernetes.io/projected/d3cfe81c-2ae4-48de-9292-cac871d9da5c-kube-api-access-f6jtb\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418728 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/df8f3888-df8e-480d-ae68-f6b16126ee83-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa31e23-7983-4949-b4cc-4fe996a91a92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-image-registry-private-configuration\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418810 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llq9d\" (UniqueName: \"kubernetes.io/projected/566fd99f-bbc7-4532-83ef-f8cce366f737-kube-api-access-llq9d\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa31e23-7983-4949-b4cc-4fe996a91a92-tmp\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2bd8\" (UniqueName: \"kubernetes.io/projected/6aa31e23-7983-4949-b4cc-4fe996a91a92-kube-api-access-m2bd8\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418876 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkn7c\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-kube-api-access-lkn7c\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418890 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa31e23-7983-4949-b4cc-4fe996a91a92-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418900 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg7bs\" (UniqueName: \"kubernetes.io/projected/894b53d3-e0f9-4c31-b48f-8d2f80edd455-kube-api-access-rg7bs\") pod \"volume-data-source-validator-78495597f6-vd82r\" (UID: \"894b53d3-e0f9-4c31-b48f-8d2f80edd455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f6dd18-06c6-4622-890d-3865aae784de-cert\") pod \"ingress-canary-6pvxk\" (UID: \"91f6dd18-06c6-4622-890d-3865aae784de\") " pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.419213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.418986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419012 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3551a07-e183-40a5-9b98-1a6865047fb9-config\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-serving-cert\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419081 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2krng\" (UniqueName: \"kubernetes.io/projected/c3551a07-e183-40a5-9b98-1a6865047fb9-kube-api-access-2krng\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419123 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa31e23-7983-4949-b4cc-4fe996a91a92-tmp\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-ca-trust-extracted\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419201 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkdgx\" (UniqueName: \"kubernetes.io/projected/df8f3888-df8e-480d-ae68-f6b16126ee83-kube-api-access-dkdgx\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419235 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzczl\" (UniqueName: \"kubernetes.io/projected/928c8c62-9201-42ec-b710-560d624ec55c-kube-api-access-hzczl\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419375 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c3551a07-e183-40a5-9b98-1a6865047fb9-trusted-ca\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-trusted-ca\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419496 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fd99f-bbc7-4532-83ef-f8cce366f737-tmp-dir\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.419913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419714 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/566fd99f-bbc7-4532-83ef-f8cce366f737-config-volume\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwzn\" (UniqueName: \"kubernetes.io/projected/980467bb-dbb4-4003-83f9-4754b8966b29-kube-api-access-tkwzn\") pod \"network-check-source-869dcfcdcd-4v8vd\" (UID: \"980467bb-dbb4-4003-83f9-4754b8966b29\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419780 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-stats-auth\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/566fd99f-bbc7-4532-83ef-f8cce366f737-tmp-dir\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419804 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-certificates\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6g5f\" (UniqueName: \"kubernetes.io/projected/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-kube-api-access-t6g5f\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-installation-pull-secrets\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-bound-sa-token\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419936 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npc29\" (UniqueName: \"kubernetes.io/projected/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-kube-api-access-npc29\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.419984 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.420003 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420013 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-config\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.420065 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:28.920046889 +0000 UTC m=+37.602525341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : secret "router-metrics-certs-default" not found Jul 12 19:22:28.420547 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420125 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6p4z\" (UniqueName: \"kubernetes.io/projected/91f6dd18-06c6-4622-890d-3865aae784de-kube-api-access-s6p4z\") pod \"ingress-canary-6pvxk\" (UID: \"91f6dd18-06c6-4622-890d-3865aae784de\") " pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.420164 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.420246 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls podName:d3cfe81c-2ae4-48de-9292-cac871d9da5c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:28.920229487 +0000 UTC m=+37.602707939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls") pod "cluster-samples-operator-65db858894-blrs8" (UID: "d3cfe81c-2ae4-48de-9292-cac871d9da5c") : secret "samples-operator-tls" not found Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420420 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-certificates\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420643 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-config\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420966 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/566fd99f-bbc7-4532-83ef-f8cce366f737-config-volume\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.421160 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.421091 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa31e23-7983-4949-b4cc-4fe996a91a92-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.421572 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.421381 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:28.921364562 +0000 UTC m=+37.603843022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : configmap references non-existent config key: service-ca.crt Jul 12 19:22:28.422110 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.421775 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:28.422110 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.421818 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls podName:df8f3888-df8e-480d-ae68-f6b16126ee83 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:28.921805838 +0000 UTC m=+37.604284275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-85cdb" (UID: "df8f3888-df8e-480d-ae68-f6b16126ee83") : secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:28.422110 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.421877 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-ca-trust-extracted\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.422110 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.420087 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c3551a07-e183-40a5-9b98-1a6865047fb9-config\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.422110 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.421990 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:22:28.422110 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.422005 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567dd678fc-ppt7c: secret "image-registry-tls" not found Jul 12 19:22:28.422110 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.422076 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls podName:a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:28.922058714 +0000 UTC m=+37.604537168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls") pod "image-registry-567dd678fc-ppt7c" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8") : secret "image-registry-tls" not found Jul 12 19:22:28.422528 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.422138 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:28.422693 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.422679 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls podName:566fd99f-bbc7-4532-83ef-f8cce366f737 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:28.922663318 +0000 UTC m=+37.605141756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls") pod "dns-default-lmpsl" (UID: "566fd99f-bbc7-4532-83ef-f8cce366f737") : secret "dns-default-metrics-tls" not found Jul 12 19:22:28.423491 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.423136 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-image-registry-private-configuration\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.423491 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.423453 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-default-certificate\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.423639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.423499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3551a07-e183-40a5-9b98-1a6865047fb9-serving-cert\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.424224 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.424161 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/91f6dd18-06c6-4622-890d-3865aae784de-cert\") pod \"ingress-canary-6pvxk\" (UID: \"91f6dd18-06c6-4622-890d-3865aae784de\") " pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.424550 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.424509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.424550 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.424530 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-installation-pull-secrets\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.424746 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.424705 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-serving-cert\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.425040 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.425015 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6aa31e23-7983-4949-b4cc-4fe996a91a92-serving-cert\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.426148 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.425711 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-stats-auth\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.428231 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.428206 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg7bs\" (UniqueName: \"kubernetes.io/projected/894b53d3-e0f9-4c31-b48f-8d2f80edd455-kube-api-access-rg7bs\") pod \"volume-data-source-validator-78495597f6-vd82r\" (UID: \"894b53d3-e0f9-4c31-b48f-8d2f80edd455\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" Jul 12 19:22:28.428534 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.428508 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jtb\" (UniqueName: \"kubernetes.io/projected/d3cfe81c-2ae4-48de-9292-cac871d9da5c-kube-api-access-f6jtb\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:28.429075 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.429052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzczl\" (UniqueName: \"kubernetes.io/projected/928c8c62-9201-42ec-b710-560d624ec55c-kube-api-access-hzczl\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.429249 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.429196 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2bd8\" (UniqueName: \"kubernetes.io/projected/6aa31e23-7983-4949-b4cc-4fe996a91a92-kube-api-access-m2bd8\") pod \"insights-operator-68d9dd94cf-52s44\" (UID: \"6aa31e23-7983-4949-b4cc-4fe996a91a92\") " pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.429866 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.429837 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkn7c\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-kube-api-access-lkn7c\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.430285 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.430262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkdgx\" (UniqueName: \"kubernetes.io/projected/df8f3888-df8e-480d-ae68-f6b16126ee83-kube-api-access-dkdgx\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.430735 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.430713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwzn\" (UniqueName: \"kubernetes.io/projected/980467bb-dbb4-4003-83f9-4754b8966b29-kube-api-access-tkwzn\") pod \"network-check-source-869dcfcdcd-4v8vd\" (UID: \"980467bb-dbb4-4003-83f9-4754b8966b29\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" Jul 12 19:22:28.431506 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.431466 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6p4z\" (UniqueName: \"kubernetes.io/projected/91f6dd18-06c6-4622-890d-3865aae784de-kube-api-access-s6p4z\") pod \"ingress-canary-6pvxk\" (UID: \"91f6dd18-06c6-4622-890d-3865aae784de\") " pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.431759 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.431738 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-bound-sa-token\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.431921 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.431901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2krng\" (UniqueName: \"kubernetes.io/projected/c3551a07-e183-40a5-9b98-1a6865047fb9-kube-api-access-2krng\") pod \"console-operator-7866497494-lcxrh\" (UID: \"c3551a07-e183-40a5-9b98-1a6865047fb9\") " pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.432201 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.432138 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npc29\" (UniqueName: \"kubernetes.io/projected/9e2e871f-6a10-44ae-a8d4-940e3c4e6489-kube-api-access-npc29\") pod \"service-ca-operator-7695fc6c8c-228q7\" (UID: \"9e2e871f-6a10-44ae-a8d4-940e3c4e6489\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.432482 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.432441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6g5f\" (UniqueName: \"kubernetes.io/projected/d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac-kube-api-access-t6g5f\") pod \"kube-storage-version-migrator-operator-799cb7f958-vjfb7\" (UID: \"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.432866 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.432846 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llq9d\" (UniqueName: \"kubernetes.io/projected/566fd99f-bbc7-4532-83ef-f8cce366f737-kube-api-access-llq9d\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.487831 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.487804 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" Jul 12 19:22:28.496157 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.496135 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:28.504489 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.504464 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" Jul 12 19:22:28.511752 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.511732 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" Jul 12 19:22:28.518780 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.518763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-52s44" Jul 12 19:22:28.524866 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.524850 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" Jul 12 19:22:28.545965 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.545945 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6pvxk" Jul 12 19:22:28.856439 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.856367 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:28.858809 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.858791 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:22:28.858933 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.858860 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dcr6j\"" Jul 12 19:22:28.923901 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.923868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.924054 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.923918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:28.924054 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.923957 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:28.924054 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.924019 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:28.924054 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924024 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.924059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924083 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:29.924065306 +0000 UTC m=+38.606543757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : secret "router-metrics-certs-default" not found Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:28.924110 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924128 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:29.924117393 +0000 UTC m=+38.606595843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : configmap references non-existent config key: service-ca.crt Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924168 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924214 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls podName:566fd99f-bbc7-4532-83ef-f8cce366f737 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:29.924203731 +0000 UTC m=+38.606682183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls") pod "dns-default-lmpsl" (UID: "566fd99f-bbc7-4532-83ef-f8cce366f737") : secret "dns-default-metrics-tls" not found Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924247 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 12 19:22:28.924261 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924263 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:28.924599 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924279 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls podName:d3cfe81c-2ae4-48de-9292-cac871d9da5c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:29.924269296 +0000 UTC m=+38.606747744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls") pod "cluster-samples-operator-65db858894-blrs8" (UID: "d3cfe81c-2ae4-48de-9292-cac871d9da5c") : secret "samples-operator-tls" not found Jul 12 19:22:28.924599 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924305 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls podName:df8f3888-df8e-480d-ae68-f6b16126ee83 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:29.924284777 +0000 UTC m=+38.606763229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-85cdb" (UID: "df8f3888-df8e-480d-ae68-f6b16126ee83") : secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:28.924599 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924343 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:22:28.924599 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924363 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567dd678fc-ppt7c: secret "image-registry-tls" not found Jul 12 19:22:28.924599 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:28.924414 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls podName:a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:29.924397541 +0000 UTC m=+38.606876000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls") pod "image-registry-567dd678fc-ppt7c" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8") : secret "image-registry-tls" not found Jul 12 19:22:29.855899 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.855864 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:29.856115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.856092 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:29.858269 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.858235 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:22:29.858770 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.858307 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cl9vq\"" Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.932964 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.933015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.933052 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.933107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.933146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:29.933187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.933333 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:29.933654 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.933392 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls podName:566fd99f-bbc7-4532-83ef-f8cce366f737 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:31.933375404 +0000 UTC m=+40.615853857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls") pod "dns-default-lmpsl" (UID: "566fd99f-bbc7-4532-83ef-f8cce366f737") : secret "dns-default-metrics-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.933746 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:31.93372639 +0000 UTC m=+40.616204830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : configmap references non-existent config key: service-ca.crt Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.933826 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.933833 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.934096 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567dd678fc-ppt7c: secret "image-registry-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.934169 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls podName:a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:31.934153389 +0000 UTC m=+40.616631850 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls") pod "image-registry-567dd678fc-ppt7c" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8") : secret "image-registry-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.934305 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls podName:df8f3888-df8e-480d-ae68-f6b16126ee83 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:31.934291678 +0000 UTC m=+40.616770121 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-85cdb" (UID: "df8f3888-df8e-480d-ae68-f6b16126ee83") : secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.933899 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.934488 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls podName:d3cfe81c-2ae4-48de-9292-cac871d9da5c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:31.934475654 +0000 UTC m=+40.616954106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls") pod "cluster-samples-operator-65db858894-blrs8" (UID: "d3cfe81c-2ae4-48de-9292-cac871d9da5c") : secret "samples-operator-tls" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.933953 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:22:29.934986 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:29.934667 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:31.934575175 +0000 UTC m=+40.617053627 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : secret "router-metrics-certs-default" not found Jul 12 19:22:30.184064 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.183793 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd"] Jul 12 19:22:30.190096 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.190067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7"] Jul 12 19:22:30.192797 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.192770 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r"] Jul 12 19:22:30.197197 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.196309 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7"] Jul 12 19:22:30.198335 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:30.198305 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f6fff9_f604_4780_bbf0_4a1f6e6fc9ac.slice/crio-c29e9db51d73afdbe5146883a5bbbf6cd94658cf051d8f7e134f5ec7e5bedaac WatchSource:0}: Error finding container c29e9db51d73afdbe5146883a5bbbf6cd94658cf051d8f7e134f5ec7e5bedaac: Status 404 returned error can't find the container with id c29e9db51d73afdbe5146883a5bbbf6cd94658cf051d8f7e134f5ec7e5bedaac Jul 12 19:22:30.199203 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:30.199165 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894b53d3_e0f9_4c31_b48f_8d2f80edd455.slice/crio-f5c8d08912b552eeeee0a46196cda9f73d67102470c26011e9ae3fabdeb988e4 WatchSource:0}: Error finding container f5c8d08912b552eeeee0a46196cda9f73d67102470c26011e9ae3fabdeb988e4: Status 404 returned error can't find the container with id f5c8d08912b552eeeee0a46196cda9f73d67102470c26011e9ae3fabdeb988e4 Jul 12 19:22:30.199833 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:30.199807 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2e871f_6a10_44ae_a8d4_940e3c4e6489.slice/crio-ee1e5c959d5148d2c4a234f5a9bea06ae72a74eec063ac440ca6e265ca3ab127 WatchSource:0}: Error finding container ee1e5c959d5148d2c4a234f5a9bea06ae72a74eec063ac440ca6e265ca3ab127: Status 404 returned error can't find the container with id ee1e5c959d5148d2c4a234f5a9bea06ae72a74eec063ac440ca6e265ca3ab127 Jul 12 19:22:30.203523 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.203413 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-lcxrh"] Jul 12 19:22:30.205822 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.205800 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6pvxk"] Jul 12 19:22:30.208936 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.208884 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-52s44"] Jul 12 19:22:30.212501 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:30.212464 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f6dd18_06c6_4622_890d_3865aae784de.slice/crio-1f3161288e2be6488fb682b84d69b7d20c19331b2f8f0ffd6dd6b04daf7a6641 WatchSource:0}: Error finding container 1f3161288e2be6488fb682b84d69b7d20c19331b2f8f0ffd6dd6b04daf7a6641: Status 404 returned error can't find the container with id 1f3161288e2be6488fb682b84d69b7d20c19331b2f8f0ffd6dd6b04daf7a6641 Jul 12 19:22:30.213448 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:30.213428 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa31e23_7983_4949_b4cc_4fe996a91a92.slice/crio-cdb42f51134d301821d9eadfc05dfddeff008da8f14ab219a244b4cf26e7dc83 WatchSource:0}: Error finding container cdb42f51134d301821d9eadfc05dfddeff008da8f14ab219a244b4cf26e7dc83: Status 404 returned error can't find the container with id cdb42f51134d301821d9eadfc05dfddeff008da8f14ab219a244b4cf26e7dc83 Jul 12 19:22:30.214136 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:30.214066 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3551a07_e183_40a5_9b98_1a6865047fb9.slice/crio-0cad4e105b4812deccbc94b0b1a9b00e7dcbbee95f571da139cfdc0f1825f7cb WatchSource:0}: Error finding container 0cad4e105b4812deccbc94b0b1a9b00e7dcbbee95f571da139cfdc0f1825f7cb: Status 404 returned error can't find the container with id 0cad4e105b4812deccbc94b0b1a9b00e7dcbbee95f571da139cfdc0f1825f7cb Jul 12 19:22:30.733638 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.733599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" event={"ID":"9e2e871f-6a10-44ae-a8d4-940e3c4e6489","Type":"ContainerStarted","Data":"ee1e5c959d5148d2c4a234f5a9bea06ae72a74eec063ac440ca6e265ca3ab127"} Jul 12 19:22:30.735651 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.735584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" event={"ID":"894b53d3-e0f9-4c31-b48f-8d2f80edd455","Type":"ContainerStarted","Data":"f5c8d08912b552eeeee0a46196cda9f73d67102470c26011e9ae3fabdeb988e4"} Jul 12 19:22:30.737062 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.737035 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-lcxrh" event={"ID":"c3551a07-e183-40a5-9b98-1a6865047fb9","Type":"ContainerStarted","Data":"0cad4e105b4812deccbc94b0b1a9b00e7dcbbee95f571da139cfdc0f1825f7cb"} Jul 12 19:22:30.739442 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.739415 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6pvxk" event={"ID":"91f6dd18-06c6-4622-890d-3865aae784de","Type":"ContainerStarted","Data":"1f3161288e2be6488fb682b84d69b7d20c19331b2f8f0ffd6dd6b04daf7a6641"} Jul 12 19:22:30.742416 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.742343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-52s44" event={"ID":"6aa31e23-7983-4949-b4cc-4fe996a91a92","Type":"ContainerStarted","Data":"cdb42f51134d301821d9eadfc05dfddeff008da8f14ab219a244b4cf26e7dc83"} Jul 12 19:22:30.744486 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.744425 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" event={"ID":"980467bb-dbb4-4003-83f9-4754b8966b29","Type":"ContainerStarted","Data":"0748dca4318e839201acec224e513d278148f4930269c19372f39fd5e09f3e2d"} Jul 12 19:22:30.746150 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.746126 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" event={"ID":"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac","Type":"ContainerStarted","Data":"c29e9db51d73afdbe5146883a5bbbf6cd94658cf051d8f7e134f5ec7e5bedaac"} Jul 12 19:22:30.751278 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.750165 2565 generic.go:356] "Generic (PLEG): container finished" podID="9710beb1-c58c-4188-92b1-61ebc42865e2" containerID="2157462c6a84dd20cbeada446cae8640ff9a0e3ebe569bcb70765859aa511dc9" exitCode=0 Jul 12 19:22:30.751450 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:30.751370 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerDied","Data":"2157462c6a84dd20cbeada446cae8640ff9a0e3ebe569bcb70765859aa511dc9"} Jul 12 19:22:31.763777 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.763740 2565 generic.go:356] "Generic (PLEG): container finished" podID="9710beb1-c58c-4188-92b1-61ebc42865e2" containerID="fb742470ec7d2f49c0ccfa80671180992ad846915c5e927957cd27c9503354d6" exitCode=0 Jul 12 19:22:31.764458 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.763824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerDied","Data":"fb742470ec7d2f49c0ccfa80671180992ad846915c5e927957cd27c9503354d6"} Jul 12 19:22:31.959391 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.959323 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:31.959391 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.959392 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:31.959619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.959421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:31.959619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.959497 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:31.959619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.959527 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:31.959619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:31.959562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:31.959861 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959761 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:31.959861 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959792 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:22:31.959861 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959816 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567dd678fc-ppt7c: secret "image-registry-tls" not found Jul 12 19:22:31.959861 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959857 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 12 19:22:31.960045 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959855 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:22:31.960045 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959818 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls podName:566fd99f-bbc7-4532-83ef-f8cce366f737 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:35.95979996 +0000 UTC m=+44.642278413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls") pod "dns-default-lmpsl" (UID: "566fd99f-bbc7-4532-83ef-f8cce366f737") : secret "dns-default-metrics-tls" not found Jul 12 19:22:31.960045 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959929 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:35.959915793 +0000 UTC m=+44.642394234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : configmap references non-existent config key: service-ca.crt Jul 12 19:22:31.960045 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959946 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls podName:a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:35.959935403 +0000 UTC m=+44.642413843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls") pod "image-registry-567dd678fc-ppt7c" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8") : secret "image-registry-tls" not found Jul 12 19:22:31.960045 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959964 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls podName:d3cfe81c-2ae4-48de-9292-cac871d9da5c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:35.959956659 +0000 UTC m=+44.642435100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls") pod "cluster-samples-operator-65db858894-blrs8" (UID: "d3cfe81c-2ae4-48de-9292-cac871d9da5c") : secret "samples-operator-tls" not found Jul 12 19:22:31.960045 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.959979 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:35.959970504 +0000 UTC m=+44.642448943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : secret "router-metrics-certs-default" not found Jul 12 19:22:31.960449 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.960266 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:31.960449 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:31.960326 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls podName:df8f3888-df8e-480d-ae68-f6b16126ee83 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:35.960310429 +0000 UTC m=+44.642788882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-85cdb" (UID: "df8f3888-df8e-480d-ae68-f6b16126ee83") : secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:32.774200 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:32.773838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dwllz" event={"ID":"9710beb1-c58c-4188-92b1-61ebc42865e2","Type":"ContainerStarted","Data":"692748ac2900199ea206f4ceef6d6a7b41743df4262ece3a380c109e0a2bc27a"} Jul 12 19:22:32.776234 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:32.776191 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" event={"ID":"894b53d3-e0f9-4c31-b48f-8d2f80edd455","Type":"ContainerStarted","Data":"a5f74dd4bc5e1cd436b90251182236ccd39619023f5fed698fe4d9af4b0b2b1f"} Jul 12 19:22:32.798863 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:32.798396 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dwllz" podStartSLOduration=10.660810035 podStartE2EDuration="40.798375089s" totalImagesPullingTime="30.137565054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:32.795682244 +0000 UTC m=+41.478160708" watchObservedRunningTime="2026-07-12 19:22:32.798375089 +0000 UTC m=+41.480853550" Jul 12 19:22:32.811624 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:32.811559 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-vd82r" podStartSLOduration=27.677338443 podStartE2EDuration="29.811537233s" totalImagesPullingTime="2.13419879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:32.81053836 +0000 UTC m=+41.493016821" watchObservedRunningTime="2026-07-12 19:22:32.811537233 +0000 UTC m=+41.494015696" Jul 12 19:22:36.001402 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:36.001364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:36.001422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:36.001448 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:36.001517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001535 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:36.001547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001558 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567dd678fc-ppt7c: secret "image-registry-tls" not found Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:36.001582 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001623 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls podName:a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:44.001601052 +0000 UTC m=+52.684079509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls") pod "image-registry-567dd678fc-ppt7c" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8") : secret "image-registry-tls" not found Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001639 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001655 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:44.001636974 +0000 UTC m=+52.684115434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : configmap references non-existent config key: service-ca.crt Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001668 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001693 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls podName:566fd99f-bbc7-4532-83ef-f8cce366f737 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:44.001677299 +0000 UTC m=+52.684155745 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls") pod "dns-default-lmpsl" (UID: "566fd99f-bbc7-4532-83ef-f8cce366f737") : secret "dns-default-metrics-tls" not found Jul 12 19:22:36.002072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001692 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:22:36.002565 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001717 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls podName:df8f3888-df8e-480d-ae68-f6b16126ee83 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:44.001707345 +0000 UTC m=+52.684185789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-85cdb" (UID: "df8f3888-df8e-480d-ae68-f6b16126ee83") : secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:36.002565 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001702 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 12 19:22:36.002565 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001749 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:44.001734114 +0000 UTC m=+52.684212551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : secret "router-metrics-certs-default" not found Jul 12 19:22:36.002565 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:36.001796 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls podName:d3cfe81c-2ae4-48de-9292-cac871d9da5c nodeName:}" failed. No retries permitted until 2026-07-12 19:22:44.001778094 +0000 UTC m=+52.684256537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls") pod "cluster-samples-operator-65db858894-blrs8" (UID: "d3cfe81c-2ae4-48de-9292-cac871d9da5c") : secret "samples-operator-tls" not found Jul 12 19:22:38.794707 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.794670 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" event={"ID":"980467bb-dbb4-4003-83f9-4754b8966b29","Type":"ContainerStarted","Data":"ec8cc86aad13eb0d8eb404160955865ebb70988e56f64b8b0db53d6de3e38190"} Jul 12 19:22:38.796660 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.796632 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" event={"ID":"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac","Type":"ContainerStarted","Data":"9d7115066223d979b00f0d2149ffc2b2444ed7a941bdb096186744301403ff9e"} Jul 12 19:22:38.798333 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.798287 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" event={"ID":"9e2e871f-6a10-44ae-a8d4-940e3c4e6489","Type":"ContainerStarted","Data":"63b91e803ad7c3a84cd3b5b63683f240360180a71b4b25200b782182f8d872f8"} Jul 12 19:22:38.800035 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.800016 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/0.log" Jul 12 19:22:38.800138 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.800055 2565 generic.go:356] "Generic (PLEG): container finished" podID="c3551a07-e183-40a5-9b98-1a6865047fb9" containerID="f1c0e4c1123a2edab53abe2b480fb2508dfec2b9b350f1f6113c16589341f90e" exitCode=255 Jul 12 19:22:38.800138 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.800121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-lcxrh" event={"ID":"c3551a07-e183-40a5-9b98-1a6865047fb9","Type":"ContainerDied","Data":"f1c0e4c1123a2edab53abe2b480fb2508dfec2b9b350f1f6113c16589341f90e"} Jul 12 19:22:38.800344 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.800327 2565 scope.go:117] "RemoveContainer" containerID="f1c0e4c1123a2edab53abe2b480fb2508dfec2b9b350f1f6113c16589341f90e" Jul 12 19:22:38.802231 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.802210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6pvxk" event={"ID":"91f6dd18-06c6-4622-890d-3865aae784de","Type":"ContainerStarted","Data":"7e1ae1553c0f1104a313887a10a8ef32e5eb383f7339863634c8a51f97942e88"} Jul 12 19:22:38.811112 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.810244 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-4v8vd" podStartSLOduration=18.023959376 podStartE2EDuration="25.81022658s" totalImagesPullingTime="7.786267204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.808667197 +0000 UTC m=+47.491145659" watchObservedRunningTime="2026-07-12 19:22:38.81022658 +0000 UTC m=+47.492705044" Jul 12 19:22:38.816482 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.816458 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-52s44" event={"ID":"6aa31e23-7983-4949-b4cc-4fe996a91a92","Type":"ContainerStarted","Data":"468bb710fcab24a19745ad33e33c30bc17edc012054d365d03353f5f7dd88c3c"} Jul 12 19:22:38.823742 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.823615 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" podStartSLOduration=28.053633171 podStartE2EDuration="35.823598629s" totalImagesPullingTime="7.769965458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.823373581 +0000 UTC m=+47.505852041" watchObservedRunningTime="2026-07-12 19:22:38.823598629 +0000 UTC m=+47.506077088" Jul 12 19:22:38.842270 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.842223 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" podStartSLOduration=28.074363276 podStartE2EDuration="35.842204236s" totalImagesPullingTime="7.76784096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.841869026 +0000 UTC m=+47.524347488" watchObservedRunningTime="2026-07-12 19:22:38.842204236 +0000 UTC m=+47.524682691" Jul 12 19:22:38.876133 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.876079 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6pvxk" podStartSLOduration=3.120102418 podStartE2EDuration="10.876059106s" totalImagesPullingTime="7.755956688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.875474479 +0000 UTC m=+47.557952939" watchObservedRunningTime="2026-07-12 19:22:38.876059106 +0000 UTC m=+47.558537567" Jul 12 19:22:38.890507 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:38.890447 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-52s44" podStartSLOduration=38.136046205 podStartE2EDuration="45.890428152s" totalImagesPullingTime="7.754381947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:38.889719002 +0000 UTC m=+47.572197463" watchObservedRunningTime="2026-07-12 19:22:38.890428152 +0000 UTC m=+47.572906613" Jul 12 19:22:39.056933 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.056862 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-5cb5t"] Jul 12 19:22:39.073220 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.073195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-5cb5t"] Jul 12 19:22:39.073350 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.073306 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:39.075885 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.075862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 12 19:22:39.076395 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.076095 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 12 19:22:39.076395 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.075958 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zpkbl\"" Jul 12 19:22:39.131238 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.131206 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d336bda-ff92-4cd6-be90-8dba32805f55-nginx-conf\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:39.131392 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.131278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:39.232093 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.232041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d336bda-ff92-4cd6-be90-8dba32805f55-nginx-conf\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:39.232304 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.232134 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:39.232304 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:39.232258 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:22:39.232409 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:39.232324 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert podName:7d336bda-ff92-4cd6-be90-8dba32805f55 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:39.732306621 +0000 UTC m=+48.414785079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-5cb5t" (UID: "7d336bda-ff92-4cd6-be90-8dba32805f55") : secret "networking-console-plugin-cert" not found Jul 12 19:22:39.232922 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.232899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7d336bda-ff92-4cd6-be90-8dba32805f55-nginx-conf\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:39.737336 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.737291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:39.737536 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:39.737464 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:22:39.737590 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:39.737547 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert podName:7d336bda-ff92-4cd6-be90-8dba32805f55 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:40.737524958 +0000 UTC m=+49.420003410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-5cb5t" (UID: "7d336bda-ff92-4cd6-be90-8dba32805f55") : secret "networking-console-plugin-cert" not found Jul 12 19:22:39.817369 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.817338 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j"] Jul 12 19:22:39.821430 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.821411 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:22:39.821865 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.821842 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/0.log" Jul 12 19:22:39.821947 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.821889 2565 generic.go:356] "Generic (PLEG): container finished" podID="c3551a07-e183-40a5-9b98-1a6865047fb9" containerID="674c4198ce8824ab54cbcab70133d7ebc675d1b566e12c3276a1c2b4eb2e5332" exitCode=255 Jul 12 19:22:39.822748 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.822725 2565 scope.go:117] "RemoveContainer" containerID="674c4198ce8824ab54cbcab70133d7ebc675d1b566e12c3276a1c2b4eb2e5332" Jul 12 19:22:39.822913 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:39.822886 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-lcxrh_openshift-console-operator(c3551a07-e183-40a5-9b98-1a6865047fb9)\"" pod="openshift-console-operator/console-operator-7866497494-lcxrh" podUID="c3551a07-e183-40a5-9b98-1a6865047fb9" Jul 12 19:22:39.847882 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.847844 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-lcxrh" event={"ID":"c3551a07-e183-40a5-9b98-1a6865047fb9","Type":"ContainerDied","Data":"674c4198ce8824ab54cbcab70133d7ebc675d1b566e12c3276a1c2b4eb2e5332"} Jul 12 19:22:39.848046 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.847900 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j"] Jul 12 19:22:39.848046 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.847950 2565 scope.go:117] "RemoveContainer" containerID="f1c0e4c1123a2edab53abe2b480fb2508dfec2b9b350f1f6113c16589341f90e" Jul 12 19:22:39.848046 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.847982 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" Jul 12 19:22:39.850640 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.850619 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 12 19:22:39.850763 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.850741 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 12 19:22:39.850864 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.850620 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-m4b94\"" Jul 12 19:22:39.939759 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.939726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d45k4\" (UniqueName: \"kubernetes.io/projected/83f1366d-c723-4434-a108-1093df72c064-kube-api-access-d45k4\") pod \"migrator-7f59b54f6c-p2q5j\" (UID: \"83f1366d-c723-4434-a108-1093df72c064\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" Jul 12 19:22:39.940319 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.939875 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dm4ws"] Jul 12 19:22:39.976246 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.976207 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dm4ws"] Jul 12 19:22:39.976432 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.976351 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:39.979082 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.979058 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 12 19:22:39.979514 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.979491 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gnkpn\"" Jul 12 19:22:39.979707 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:39.979674 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 12 19:22:40.040666 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.040589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53fba168-256f-4c00-a11a-69b3a3224fff-data-volume\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.040807 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.040664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nql8f\" (UniqueName: \"kubernetes.io/projected/53fba168-256f-4c00-a11a-69b3a3224fff-kube-api-access-nql8f\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.040807 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.040690 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.040807 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.040746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d45k4\" (UniqueName: \"kubernetes.io/projected/83f1366d-c723-4434-a108-1093df72c064-kube-api-access-d45k4\") pod \"migrator-7f59b54f6c-p2q5j\" (UID: \"83f1366d-c723-4434-a108-1093df72c064\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" Jul 12 19:22:40.040929 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.040813 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.040929 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.040847 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53fba168-256f-4c00-a11a-69b3a3224fff-crio-socket\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.057681 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.057649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d45k4\" (UniqueName: \"kubernetes.io/projected/83f1366d-c723-4434-a108-1093df72c064-kube-api-access-d45k4\") pod \"migrator-7f59b54f6c-p2q5j\" (UID: \"83f1366d-c723-4434-a108-1093df72c064\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" Jul 12 19:22:40.141887 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.141851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.141887 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.141891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53fba168-256f-4c00-a11a-69b3a3224fff-crio-socket\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.142132 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.141971 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53fba168-256f-4c00-a11a-69b3a3224fff-data-volume\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.142132 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.141971 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:22:40.142132 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.142049 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nql8f\" (UniqueName: \"kubernetes.io/projected/53fba168-256f-4c00-a11a-69b3a3224fff-kube-api-access-nql8f\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.142132 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.142087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.142132 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.142099 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:40.642082802 +0000 UTC m=+49.324561244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : configmap "kube-rbac-proxy" not found Jul 12 19:22:40.142381 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.142200 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:22:40.142381 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.142259 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:40.642242354 +0000 UTC m=+49.324720808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : secret "insights-runtime-extractor-tls" not found Jul 12 19:22:40.142381 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.142300 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53fba168-256f-4c00-a11a-69b3a3224fff-crio-socket\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.142500 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.142477 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53fba168-256f-4c00-a11a-69b3a3224fff-data-volume\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.149542 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.149517 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nql8f\" (UniqueName: \"kubernetes.io/projected/53fba168-256f-4c00-a11a-69b3a3224fff-kube-api-access-nql8f\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.158248 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.158232 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" Jul 12 19:22:40.298416 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.298344 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j"] Jul 12 19:22:40.301111 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:40.301085 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f1366d_c723_4434_a108_1093df72c064.slice/crio-64a73104eced9ad867ca1cf2ddf3e381ec434ab8f34e73dd22dd67703fd3dc87 WatchSource:0}: Error finding container 64a73104eced9ad867ca1cf2ddf3e381ec434ab8f34e73dd22dd67703fd3dc87: Status 404 returned error can't find the container with id 64a73104eced9ad867ca1cf2ddf3e381ec434ab8f34e73dd22dd67703fd3dc87 Jul 12 19:22:40.646801 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.646768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.646951 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.646822 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:40.646951 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.646887 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:22:40.646951 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.646923 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:22:40.647044 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.646933 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:41.646918397 +0000 UTC m=+50.329396834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : configmap "kube-rbac-proxy" not found Jul 12 19:22:40.647105 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.647091 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:41.647071888 +0000 UTC m=+50.329550341 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : secret "insights-runtime-extractor-tls" not found Jul 12 19:22:40.748506 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.748473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:40.748648 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.748630 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:22:40.748705 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.748696 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert podName:7d336bda-ff92-4cd6-be90-8dba32805f55 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:42.748678765 +0000 UTC m=+51.431157206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-5cb5t" (UID: "7d336bda-ff92-4cd6-be90-8dba32805f55") : secret "networking-console-plugin-cert" not found Jul 12 19:22:40.826708 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.826671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" event={"ID":"83f1366d-c723-4434-a108-1093df72c064","Type":"ContainerStarted","Data":"64a73104eced9ad867ca1cf2ddf3e381ec434ab8f34e73dd22dd67703fd3dc87"} Jul 12 19:22:40.828264 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.828246 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:22:40.828619 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:40.828593 2565 scope.go:117] "RemoveContainer" containerID="674c4198ce8824ab54cbcab70133d7ebc675d1b566e12c3276a1c2b4eb2e5332" Jul 12 19:22:40.828808 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:40.828787 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-lcxrh_openshift-console-operator(c3551a07-e183-40a5-9b98-1a6865047fb9)\"" pod="openshift-console-operator/console-operator-7866497494-lcxrh" podUID="c3551a07-e183-40a5-9b98-1a6865047fb9" Jul 12 19:22:41.658391 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:41.658355 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:41.658557 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:41.658411 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:41.658557 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:41.658489 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:22:41.658557 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:41.658513 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:22:41.658557 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:41.658540 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:43.658525066 +0000 UTC m=+52.341003504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : configmap "kube-rbac-proxy" not found Jul 12 19:22:41.658730 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:41.658573 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:43.65855519 +0000 UTC m=+52.341033631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : secret "insights-runtime-extractor-tls" not found Jul 12 19:22:41.734747 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:41.734722 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n6xsf_320c816e-c163-4ccb-a5fd-a6e8d2eb88df/dns-node-resolver/0.log" Jul 12 19:22:42.419378 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.419347 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bjrc6"] Jul 12 19:22:42.438674 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.438637 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bjrc6"] Jul 12 19:22:42.438824 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.438754 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.441334 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.441311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 12 19:22:42.441334 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.441311 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 12 19:22:42.441508 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.441311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 12 19:22:42.442108 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.442094 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 12 19:22:42.442163 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.442150 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fvh5j\"" Jul 12 19:22:42.567457 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.567426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1bec3744-ce85-4cf5-b411-c808b173f463-signing-cabundle\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.567615 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.567552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1bec3744-ce85-4cf5-b411-c808b173f463-signing-key\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.567615 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.567594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wzzr\" (UniqueName: \"kubernetes.io/projected/1bec3744-ce85-4cf5-b411-c808b173f463-kube-api-access-6wzzr\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.668680 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.668646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1bec3744-ce85-4cf5-b411-c808b173f463-signing-key\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.668873 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.668697 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wzzr\" (UniqueName: \"kubernetes.io/projected/1bec3744-ce85-4cf5-b411-c808b173f463-kube-api-access-6wzzr\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.668873 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.668769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1bec3744-ce85-4cf5-b411-c808b173f463-signing-cabundle\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.669452 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.669406 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1bec3744-ce85-4cf5-b411-c808b173f463-signing-cabundle\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.671067 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.671043 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1bec3744-ce85-4cf5-b411-c808b173f463-signing-key\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.676339 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.676313 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wzzr\" (UniqueName: \"kubernetes.io/projected/1bec3744-ce85-4cf5-b411-c808b173f463-kube-api-access-6wzzr\") pod \"service-ca-79f68b9f4f-bjrc6\" (UID: \"1bec3744-ce85-4cf5-b411-c808b173f463\") " pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.748130 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.748093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" Jul 12 19:22:42.769780 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.769758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:42.769934 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:42.769915 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:22:42.770003 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:42.769993 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert podName:7d336bda-ff92-4cd6-be90-8dba32805f55 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:46.769978854 +0000 UTC m=+55.452457292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-5cb5t" (UID: "7d336bda-ff92-4cd6-be90-8dba32805f55") : secret "networking-console-plugin-cert" not found Jul 12 19:22:42.837888 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.837860 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" event={"ID":"83f1366d-c723-4434-a108-1093df72c064","Type":"ContainerStarted","Data":"5f445afe5b729f89922e7f747c297288ff311441237fc342b31ca668de611fa5"} Jul 12 19:22:42.838024 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.837913 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" event={"ID":"83f1366d-c723-4434-a108-1093df72c064","Type":"ContainerStarted","Data":"eafc6cec6a9623457981d5f5983dcacb275301721b0a281c197604ec5afd8dee"} Jul 12 19:22:42.852960 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.852916 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p2q5j" podStartSLOduration=2.260037577 podStartE2EDuration="3.852902439s" totalImagesPullingTime="1.592864862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:42.852265778 +0000 UTC m=+51.534744240" watchObservedRunningTime="2026-07-12 19:22:42.852902439 +0000 UTC m=+51.535380944" Jul 12 19:22:42.879052 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.879024 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-bjrc6"] Jul 12 19:22:42.883577 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:42.883550 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bec3744_ce85_4cf5_b411_c808b173f463.slice/crio-8c74855aa641dca839b21b00ed8308fc9bf3f516f9ac1066991820590d39300b WatchSource:0}: Error finding container 8c74855aa641dca839b21b00ed8308fc9bf3f516f9ac1066991820590d39300b: Status 404 returned error can't find the container with id 8c74855aa641dca839b21b00ed8308fc9bf3f516f9ac1066991820590d39300b Jul 12 19:22:42.938921 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:42.938900 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zgx4p_43101f96-47a6-4a66-b3bc-36244a83785b/node-ca/0.log" Jul 12 19:22:43.334954 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.334879 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6pvxk_91f6dd18-06c6-4622-890d-3865aae784de/serve-healthcheck-canary/0.log" Jul 12 19:22:43.535353 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.535325 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-p2q5j_83f1366d-c723-4434-a108-1093df72c064/migrator/0.log" Jul 12 19:22:43.677632 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.677598 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:43.677825 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.677652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:43.677825 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:43.677738 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 12 19:22:43.677825 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:43.677745 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:22:43.677825 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:43.677783 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:47.677769555 +0000 UTC m=+56.360247994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : configmap "kube-rbac-proxy" not found Jul 12 19:22:43.677825 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:43.677797 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:47.677790321 +0000 UTC m=+56.360268759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : secret "insights-runtime-extractor-tls" not found Jul 12 19:22:43.734841 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.734804 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-p2q5j_83f1366d-c723-4434-a108-1093df72c064/graceful-termination/0.log" Jul 12 19:22:43.844304 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.844265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" event={"ID":"1bec3744-ce85-4cf5-b411-c808b173f463","Type":"ContainerStarted","Data":"17cc3d9a8130cef5cd2a550d7ae54dda89097adaed1eac7cc7a59f89c450ec2b"} Jul 12 19:22:43.844495 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.844322 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" event={"ID":"1bec3744-ce85-4cf5-b411-c808b173f463","Type":"ContainerStarted","Data":"8c74855aa641dca839b21b00ed8308fc9bf3f516f9ac1066991820590d39300b"} Jul 12 19:22:43.860211 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.860146 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-bjrc6" podStartSLOduration=1.860131906 podStartE2EDuration="1.860131906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:43.859155856 +0000 UTC m=+52.541634316" watchObservedRunningTime="2026-07-12 19:22:43.860131906 +0000 UTC m=+52.542610366" Jul 12 19:22:43.940569 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:43.940463 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-vjfb7_d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac/kube-storage-version-migrator-operator/0.log" Jul 12 19:22:44.081878 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:44.081838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:22:44.082039 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:44.081889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:44.082039 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:44.081914 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:22:44.082039 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:44.082009 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:22:44.082039 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082014 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 12 19:22:44.082039 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082038 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-567dd678fc-ppt7c: secret "image-registry-tls" not found Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082026 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:23:00.082007279 +0000 UTC m=+68.764485738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : configmap references non-existent config key: service-ca.crt Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082071 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082102 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls podName:a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8 nodeName:}" failed. No retries permitted until 2026-07-12 19:23:00.082085736 +0000 UTC m=+68.764564181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls") pod "image-registry-567dd678fc-ppt7c" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8") : secret "image-registry-tls" not found Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082078 2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082120 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls podName:566fd99f-bbc7-4532-83ef-f8cce366f737 nodeName:}" failed. No retries permitted until 2026-07-12 19:23:00.082110669 +0000 UTC m=+68.764589113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls") pod "dns-default-lmpsl" (UID: "566fd99f-bbc7-4532-83ef-f8cce366f737") : secret "dns-default-metrics-tls" not found Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082137 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs podName:928c8c62-9201-42ec-b710-560d624ec55c nodeName:}" failed. No retries permitted until 2026-07-12 19:23:00.082128811 +0000 UTC m=+68.764607251 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs") pod "router-default-86d8bc5964-wgq5f" (UID: "928c8c62-9201-42ec-b710-560d624ec55c") : secret "router-metrics-certs-default" not found Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:44.082171 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:22:44.082257 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:44.082233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:22:44.082579 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082337 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 12 19:22:44.082579 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082364 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:44.082579 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082414 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls podName:d3cfe81c-2ae4-48de-9292-cac871d9da5c nodeName:}" failed. No retries permitted until 2026-07-12 19:23:00.082395716 +0000 UTC m=+68.764874170 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls") pod "cluster-samples-operator-65db858894-blrs8" (UID: "d3cfe81c-2ae4-48de-9292-cac871d9da5c") : secret "samples-operator-tls" not found Jul 12 19:22:44.082579 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:44.082435 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls podName:df8f3888-df8e-480d-ae68-f6b16126ee83 nodeName:}" failed. No retries permitted until 2026-07-12 19:23:00.082424803 +0000 UTC m=+68.764903252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-85cdb" (UID: "df8f3888-df8e-480d-ae68-f6b16126ee83") : secret "cluster-monitoring-operator-tls" not found Jul 12 19:22:46.806473 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:46.806437 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:46.806972 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:46.806583 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 12 19:22:46.806972 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:46.806649 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert podName:7d336bda-ff92-4cd6-be90-8dba32805f55 nodeName:}" failed. No retries permitted until 2026-07-12 19:22:54.806632607 +0000 UTC m=+63.489111061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-5cb5t" (UID: "7d336bda-ff92-4cd6-be90-8dba32805f55") : secret "networking-console-plugin-cert" not found Jul 12 19:22:47.713921 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:47.713882 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:47.714129 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:47.713962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:47.714129 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:47.714026 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 12 19:22:47.714129 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:47.714090 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls podName:53fba168-256f-4c00-a11a-69b3a3224fff nodeName:}" failed. No retries permitted until 2026-07-12 19:22:55.714074017 +0000 UTC m=+64.396552458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dm4ws" (UID: "53fba168-256f-4c00-a11a-69b3a3224fff") : secret "insights-runtime-extractor-tls" not found Jul 12 19:22:47.726269 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:47.726243 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fba168-256f-4c00-a11a-69b3a3224fff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:48.497218 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:48.497169 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:48.497585 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:48.497241 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:22:48.497643 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:48.497597 2565 scope.go:117] "RemoveContainer" containerID="674c4198ce8824ab54cbcab70133d7ebc675d1b566e12c3276a1c2b4eb2e5332" Jul 12 19:22:48.497762 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:22:48.497741 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-lcxrh_openshift-console-operator(c3551a07-e183-40a5-9b98-1a6865047fb9)\"" pod="openshift-console-operator/console-operator-7866497494-lcxrh" podUID="c3551a07-e183-40a5-9b98-1a6865047fb9" Jul 12 19:22:54.724760 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:54.724734 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gnfrs" Jul 12 19:22:54.876601 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:54.876560 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:54.879148 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:54.879123 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7d336bda-ff92-4cd6-be90-8dba32805f55-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-5cb5t\" (UID: \"7d336bda-ff92-4cd6-be90-8dba32805f55\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:54.986137 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:54.986069 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-zpkbl\"" Jul 12 19:22:54.993826 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:54.993808 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" Jul 12 19:22:55.138519 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:55.138381 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-5cb5t"] Jul 12 19:22:55.140913 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:55.140888 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d336bda_ff92_4cd6_be90_8dba32805f55.slice/crio-1b8d1c5c66637ebb5bfb9390470c2c0334307f997235c8268bc18d7775fd22f1 WatchSource:0}: Error finding container 1b8d1c5c66637ebb5bfb9390470c2c0334307f997235c8268bc18d7775fd22f1: Status 404 returned error can't find the container with id 1b8d1c5c66637ebb5bfb9390470c2c0334307f997235c8268bc18d7775fd22f1 Jul 12 19:22:55.784964 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:55.784933 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:55.787572 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:55.787541 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fba168-256f-4c00-a11a-69b3a3224fff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dm4ws\" (UID: \"53fba168-256f-4c00-a11a-69b3a3224fff\") " pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:55.883501 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:55.883471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" event={"ID":"7d336bda-ff92-4cd6-be90-8dba32805f55","Type":"ContainerStarted","Data":"1b8d1c5c66637ebb5bfb9390470c2c0334307f997235c8268bc18d7775fd22f1"} Jul 12 19:22:55.891081 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:55.891056 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gnkpn\"" Jul 12 19:22:55.899013 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:55.898992 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dm4ws" Jul 12 19:22:56.267408 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:56.267318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dm4ws"] Jul 12 19:22:56.269940 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:22:56.269917 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fba168_256f_4c00_a11a_69b3a3224fff.slice/crio-4d098f8750eda82915b62da3e9a183660f51c43355fb6c53fd603a38cc93dc1d WatchSource:0}: Error finding container 4d098f8750eda82915b62da3e9a183660f51c43355fb6c53fd603a38cc93dc1d: Status 404 returned error can't find the container with id 4d098f8750eda82915b62da3e9a183660f51c43355fb6c53fd603a38cc93dc1d Jul 12 19:22:56.890383 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:56.890347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm4ws" event={"ID":"53fba168-256f-4c00-a11a-69b3a3224fff","Type":"ContainerStarted","Data":"4d098f8750eda82915b62da3e9a183660f51c43355fb6c53fd603a38cc93dc1d"} Jul 12 19:22:56.892470 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:56.892437 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" event={"ID":"7d336bda-ff92-4cd6-be90-8dba32805f55","Type":"ContainerStarted","Data":"4ecf5f53d41e372be07b14d7c0730b3879ea84cd5f60a69e7138b00a2db46810"} Jul 12 19:22:56.907585 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:56.907513 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-5cb5t" podStartSLOduration=16.884363044 podStartE2EDuration="17.907494414s" totalImagesPullingTime="1.02313137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:22:56.906281649 +0000 UTC m=+65.588760110" watchObservedRunningTime="2026-07-12 19:22:56.907494414 +0000 UTC m=+65.589972875" Jul 12 19:22:59.418810 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.418776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:59.419390 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.418848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:59.421602 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.421581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 12 19:22:59.422120 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.422104 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 12 19:22:59.431586 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.431558 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8899441-0b5a-4714-9743-f03b4cc1ed86-metrics-certs\") pod \"network-metrics-daemon-ggg8r\" (UID: \"a8899441-0b5a-4714-9743-f03b4cc1ed86\") " pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:59.431715 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.431591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c57f6e17-8e69-4af8-9a1e-540b29c4874c-original-pull-secret\") pod \"global-pull-secret-syncer-bblrg\" (UID: \"c57f6e17-8e69-4af8-9a1e-540b29c4874c\") " pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:59.469686 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.469662 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dcr6j\"" Jul 12 19:22:59.478001 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.477973 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ggg8r" Jul 12 19:22:59.519968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.519943 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:59.522353 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.522335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl2dw\" (UniqueName: \"kubernetes.io/projected/444d7211-cd07-42fa-a693-e5a7c8d32d59-kube-api-access-dl2dw\") pod \"network-check-target-kpjl6\" (UID: \"444d7211-cd07-42fa-a693-e5a7c8d32d59\") " pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:59.569741 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.569714 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cl9vq\"" Jul 12 19:22:59.573842 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.573821 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bblrg" Jul 12 19:22:59.577917 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.577901 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:22:59.856252 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:22:59.856223 2565 scope.go:117] "RemoveContainer" containerID="674c4198ce8824ab54cbcab70133d7ebc675d1b566e12c3276a1c2b4eb2e5332" Jul 12 19:23:00.024540 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.024493 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bblrg"] Jul 12 19:23:00.032757 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:23:00.032714 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57f6e17_8e69_4af8_9a1e_540b29c4874c.slice/crio-59116ad1c3f4b6a5734eb0d3d0709830a70950625cfd17a9b1861d04a42b3c9a WatchSource:0}: Error finding container 59116ad1c3f4b6a5734eb0d3d0709830a70950625cfd17a9b1861d04a42b3c9a: Status 404 returned error can't find the container with id 59116ad1c3f4b6a5734eb0d3d0709830a70950625cfd17a9b1861d04a42b3c9a Jul 12 19:23:00.045250 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.045222 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ggg8r"] Jul 12 19:23:00.047120 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:23:00.047098 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8899441_0b5a_4714_9743_f03b4cc1ed86.slice/crio-c38ecec83bdc9eba332251565d2083807602f7d55a6c2df70bf9c9be2d056197 WatchSource:0}: Error finding container c38ecec83bdc9eba332251565d2083807602f7d55a6c2df70bf9c9be2d056197: Status 404 returned error can't find the container with id c38ecec83bdc9eba332251565d2083807602f7d55a6c2df70bf9c9be2d056197 Jul 12 19:23:00.061483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.061328 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kpjl6"] Jul 12 19:23:00.063826 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:23:00.063801 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444d7211_cd07_42fa_a693_e5a7c8d32d59.slice/crio-0838ca382dab635b25b095cf5ceace7ea7d4eb7bc62514465bf713527c891c14 WatchSource:0}: Error finding container 0838ca382dab635b25b095cf5ceace7ea7d4eb7bc62514465bf713527c891c14: Status 404 returned error can't find the container with id 0838ca382dab635b25b095cf5ceace7ea7d4eb7bc62514465bf713527c891c14 Jul 12 19:23:00.126856 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.126832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:00.126952 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.126867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:23:00.126952 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.126889 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:23:00.127048 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.126998 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:23:00.127048 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.127032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:00.127127 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.127058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:23:00.127941 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.127879 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/928c8c62-9201-42ec-b710-560d624ec55c-service-ca-bundle\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:00.129469 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.129452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"image-registry-567dd678fc-ppt7c\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:23:00.129654 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.129631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/d3cfe81c-2ae4-48de-9292-cac871d9da5c-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-blrs8\" (UID: \"d3cfe81c-2ae4-48de-9292-cac871d9da5c\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:23:00.129700 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.129679 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/566fd99f-bbc7-4532-83ef-f8cce366f737-metrics-tls\") pod \"dns-default-lmpsl\" (UID: \"566fd99f-bbc7-4532-83ef-f8cce366f737\") " pod="openshift-dns/dns-default-lmpsl" Jul 12 19:23:00.129758 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.129742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/df8f3888-df8e-480d-ae68-f6b16126ee83-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-85cdb\" (UID: \"df8f3888-df8e-480d-ae68-f6b16126ee83\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:23:00.129797 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.129744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928c8c62-9201-42ec-b710-560d624ec55c-metrics-certs\") pod \"router-default-86d8bc5964-wgq5f\" (UID: \"928c8c62-9201-42ec-b710-560d624ec55c\") " pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:00.271878 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.271850 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dx58x\"" Jul 12 19:23:00.279516 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.279479 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:23:00.279516 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.279511 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-bs8bw\"" Jul 12 19:23:00.288398 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.288377 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" Jul 12 19:23:00.334394 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.334151 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jt2h5\"" Jul 12 19:23:00.341387 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.341135 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4q58t\"" Jul 12 19:23:00.342266 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.342130 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:00.350471 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.350234 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" Jul 12 19:23:00.357262 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.357086 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kfwdz\"" Jul 12 19:23:00.363041 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.363012 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lmpsl" Jul 12 19:23:00.461597 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.461525 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-567dd678fc-ppt7c"] Jul 12 19:23:00.503445 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.502885 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb"] Jul 12 19:23:00.617668 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.617546 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8"] Jul 12 19:23:00.637632 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.637474 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lmpsl"] Jul 12 19:23:00.641602 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:23:00.641440 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566fd99f_bbc7_4532_83ef_f8cce366f737.slice/crio-6c9ef469893a30d6dfbf12d7bb71c78d5e2b5d4e7efbffc89942862a387091d7 WatchSource:0}: Error finding container 6c9ef469893a30d6dfbf12d7bb71c78d5e2b5d4e7efbffc89942862a387091d7: Status 404 returned error can't find the container with id 6c9ef469893a30d6dfbf12d7bb71c78d5e2b5d4e7efbffc89942862a387091d7 Jul 12 19:23:00.844445 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.844377 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86d8bc5964-wgq5f"] Jul 12 19:23:00.849734 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:23:00.849707 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928c8c62_9201_42ec_b710_560d624ec55c.slice/crio-47402d5d31a26d931eb4e8407b415d6bc539efed72aad83676063e578378043b WatchSource:0}: Error finding container 47402d5d31a26d931eb4e8407b415d6bc539efed72aad83676063e578378043b: Status 404 returned error can't find the container with id 47402d5d31a26d931eb4e8407b415d6bc539efed72aad83676063e578378043b Jul 12 19:23:00.911159 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.911129 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kpjl6" event={"ID":"444d7211-cd07-42fa-a693-e5a7c8d32d59","Type":"ContainerStarted","Data":"ff054b5d6aeb13afcabd5630cb59a010e10d3244e44680c9740e9ff9bbcddb0e"} Jul 12 19:23:00.911275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.911207 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kpjl6" event={"ID":"444d7211-cd07-42fa-a693-e5a7c8d32d59","Type":"ContainerStarted","Data":"0838ca382dab635b25b095cf5ceace7ea7d4eb7bc62514465bf713527c891c14"} Jul 12 19:23:00.911478 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.911460 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:23:00.913337 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.913272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmpsl" event={"ID":"566fd99f-bbc7-4532-83ef-f8cce366f737","Type":"ContainerStarted","Data":"6c9ef469893a30d6dfbf12d7bb71c78d5e2b5d4e7efbffc89942862a387091d7"} Jul 12 19:23:00.917023 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.915659 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" event={"ID":"df8f3888-df8e-480d-ae68-f6b16126ee83","Type":"ContainerStarted","Data":"b0fe1c75693bd00c91d14009ec256492ea00a0c16179336ad2fb8eae4fe904be"} Jul 12 19:23:00.919013 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.918986 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:23:00.919165 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.919094 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-lcxrh" event={"ID":"c3551a07-e183-40a5-9b98-1a6865047fb9","Type":"ContainerStarted","Data":"8b8e4e068e74db6fffa6d2ce8960b014706712d696a9efeb58ab2a93c47bbb9a"} Jul 12 19:23:00.919962 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.919454 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:23:00.920951 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.920924 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" event={"ID":"d3cfe81c-2ae4-48de-9292-cac871d9da5c","Type":"ContainerStarted","Data":"c6d99d0f9a0b6b5e3e53b93dc19ef74310df3baec9ac42318d1a5421f056c7e3"} Jul 12 19:23:00.926849 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.926479 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kpjl6" podStartSLOduration=68.926461629 podStartE2EDuration="1m8.926461629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:00.925108834 +0000 UTC m=+69.607587308" watchObservedRunningTime="2026-07-12 19:23:00.926461629 +0000 UTC m=+69.608940091" Jul 12 19:23:00.926849 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.926579 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-lcxrh" Jul 12 19:23:00.927363 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.927336 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm4ws" event={"ID":"53fba168-256f-4c00-a11a-69b3a3224fff","Type":"ContainerStarted","Data":"04cc1d5076d62c05275cadb2d1fdb4dd135c5b5e29f71c6b29e8c083a795d009"} Jul 12 19:23:00.929045 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.929001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" event={"ID":"928c8c62-9201-42ec-b710-560d624ec55c","Type":"ContainerStarted","Data":"47402d5d31a26d931eb4e8407b415d6bc539efed72aad83676063e578378043b"} Jul 12 19:23:00.930897 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.930837 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bblrg" event={"ID":"c57f6e17-8e69-4af8-9a1e-540b29c4874c","Type":"ContainerStarted","Data":"59116ad1c3f4b6a5734eb0d3d0709830a70950625cfd17a9b1861d04a42b3c9a"} Jul 12 19:23:00.932294 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.932226 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ggg8r" event={"ID":"a8899441-0b5a-4714-9743-f03b4cc1ed86","Type":"ContainerStarted","Data":"c38ecec83bdc9eba332251565d2083807602f7d55a6c2df70bf9c9be2d056197"} Jul 12 19:23:00.935027 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.934794 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" event={"ID":"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8","Type":"ContainerStarted","Data":"fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802"} Jul 12 19:23:00.935027 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.934839 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" event={"ID":"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8","Type":"ContainerStarted","Data":"90dfc4667791c3268a42170824f024ccd93862ca17ac26567ff0d797a189c695"} Jul 12 19:23:00.935140 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.935055 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:23:00.941866 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.940152 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-lcxrh" podStartSLOduration=50.186112935 podStartE2EDuration="57.940139582s" totalImagesPullingTime="7.754026647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:00.939495633 +0000 UTC m=+69.621974087" watchObservedRunningTime="2026-07-12 19:23:00.940139582 +0000 UTC m=+69.622618042" Jul 12 19:23:00.971369 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:00.971315 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" podStartSLOduration=67.971295211 podStartE2EDuration="1m7.971295211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:00.971030627 +0000 UTC m=+69.653509102" watchObservedRunningTime="2026-07-12 19:23:00.971295211 +0000 UTC m=+69.653773671" Jul 12 19:23:01.234408 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.234331 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4"] Jul 12 19:23:01.252473 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.252427 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4"] Jul 12 19:23:01.252912 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.252827 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.254607 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.254588 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c"] Jul 12 19:23:01.256202 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.255933 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 12 19:23:01.256309 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.256212 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 12 19:23:01.256381 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.256358 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 12 19:23:01.256501 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.256441 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 12 19:23:01.265695 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.264933 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j"] Jul 12 19:23:01.265695 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.265153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.267752 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.267733 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 12 19:23:01.268104 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.268086 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 12 19:23:01.268574 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.268556 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 12 19:23:01.269670 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.269648 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 12 19:23:01.286725 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.286654 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j"] Jul 12 19:23:01.286725 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.286689 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c"] Jul 12 19:23:01.287492 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.287049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.289520 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.289499 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 12 19:23:01.289771 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.289755 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5wtq6\"" Jul 12 19:23:01.328227 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.328173 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-f6ng5"] Jul 12 19:23:01.333113 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.333090 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-f6ng5" Jul 12 19:23:01.335709 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.335688 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 12 19:23:01.336094 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.335966 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-m2vbf\"" Jul 12 19:23:01.336094 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.335980 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 12 19:23:01.342470 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342395 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-f6ng5"] Jul 12 19:23:01.342571 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsbcq\" (UniqueName: \"kubernetes.io/projected/5e13f450-f8cd-43aa-9864-9670f84317ec-kube-api-access-bsbcq\") pod \"downloads-68c9cbb497-f6ng5\" (UID: \"5e13f450-f8cd-43aa-9864-9670f84317ec\") " pod="openshift-console/downloads-68c9cbb497-f6ng5" Jul 12 19:23:01.342571 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65829e68-acff-4d76-9fc0-3907abadd232-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.342691 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342579 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84spz\" (UniqueName: \"kubernetes.io/projected/65829e68-acff-4d76-9fc0-3907abadd232-kube-api-access-84spz\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.342691 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342606 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-hub\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.342691 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05cec323-9469-437a-940d-1c370643767f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8586898f64-hrs8j\" (UID: \"05cec323-9469-437a-940d-1c370643767f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.342691 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.342891 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.342891 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7197c1de-7810-4eba-b11a-71b70e4c090d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.342891 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x2vq\" (UniqueName: \"kubernetes.io/projected/05cec323-9469-437a-940d-1c370643767f-kube-api-access-7x2vq\") pod \"managed-serviceaccount-addon-agent-8586898f64-hrs8j\" (UID: \"05cec323-9469-437a-940d-1c370643767f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.342891 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-ca\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.342891 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t6zj\" (UniqueName: \"kubernetes.io/projected/7197c1de-7810-4eba-b11a-71b70e4c090d-kube-api-access-2t6zj\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.342891 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.342860 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65829e68-acff-4d76-9fc0-3907abadd232-tmp\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.443621 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443586 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t6zj\" (UniqueName: \"kubernetes.io/projected/7197c1de-7810-4eba-b11a-71b70e4c090d-kube-api-access-2t6zj\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.443793 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443635 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65829e68-acff-4d76-9fc0-3907abadd232-tmp\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.443793 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsbcq\" (UniqueName: \"kubernetes.io/projected/5e13f450-f8cd-43aa-9864-9670f84317ec-kube-api-access-bsbcq\") pod \"downloads-68c9cbb497-f6ng5\" (UID: \"5e13f450-f8cd-43aa-9864-9670f84317ec\") " pod="openshift-console/downloads-68c9cbb497-f6ng5" Jul 12 19:23:01.443793 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65829e68-acff-4d76-9fc0-3907abadd232-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.443793 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84spz\" (UniqueName: \"kubernetes.io/projected/65829e68-acff-4d76-9fc0-3907abadd232-kube-api-access-84spz\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.443793 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443757 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-hub\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.443793 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05cec323-9469-437a-940d-1c370643767f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8586898f64-hrs8j\" (UID: \"05cec323-9469-437a-940d-1c370643767f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.444292 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443810 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.444292 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.444292 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7197c1de-7810-4eba-b11a-71b70e4c090d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.444292 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443912 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x2vq\" (UniqueName: \"kubernetes.io/projected/05cec323-9469-437a-940d-1c370643767f-kube-api-access-7x2vq\") pod \"managed-serviceaccount-addon-agent-8586898f64-hrs8j\" (UID: \"05cec323-9469-437a-940d-1c370643767f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.444292 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.443947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-ca\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.444559 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.444528 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65829e68-acff-4d76-9fc0-3907abadd232-tmp\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.445011 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.444989 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7197c1de-7810-4eba-b11a-71b70e4c090d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.447746 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.447721 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.447855 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.447781 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-hub\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.448474 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.448381 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05cec323-9469-437a-940d-1c370643767f-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8586898f64-hrs8j\" (UID: \"05cec323-9469-437a-940d-1c370643767f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.448474 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.448423 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-ca\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.450070 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.450044 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7197c1de-7810-4eba-b11a-71b70e4c090d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.450722 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.450699 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/65829e68-acff-4d76-9fc0-3907abadd232-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.453526 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.453481 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t6zj\" (UniqueName: \"kubernetes.io/projected/7197c1de-7810-4eba-b11a-71b70e4c090d-kube-api-access-2t6zj\") pod \"cluster-proxy-proxy-agent-5896db89db-fnd9c\" (UID: \"7197c1de-7810-4eba-b11a-71b70e4c090d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.453961 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.453902 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84spz\" (UniqueName: \"kubernetes.io/projected/65829e68-acff-4d76-9fc0-3907abadd232-kube-api-access-84spz\") pod \"klusterlet-addon-workmgr-7fb4668789-8wtk4\" (UID: \"65829e68-acff-4d76-9fc0-3907abadd232\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.454710 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.454670 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x2vq\" (UniqueName: \"kubernetes.io/projected/05cec323-9469-437a-940d-1c370643767f-kube-api-access-7x2vq\") pod \"managed-serviceaccount-addon-agent-8586898f64-hrs8j\" (UID: \"05cec323-9469-437a-940d-1c370643767f\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.455553 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.455531 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsbcq\" (UniqueName: \"kubernetes.io/projected/5e13f450-f8cd-43aa-9864-9670f84317ec-kube-api-access-bsbcq\") pod \"downloads-68c9cbb497-f6ng5\" (UID: \"5e13f450-f8cd-43aa-9864-9670f84317ec\") " pod="openshift-console/downloads-68c9cbb497-f6ng5" Jul 12 19:23:01.570503 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.570312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:01.590197 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.590153 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" Jul 12 19:23:01.600712 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.600686 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" Jul 12 19:23:01.656088 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.656059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-f6ng5" Jul 12 19:23:01.771662 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.771580 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4"] Jul 12 19:23:01.785512 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:23:01.785474 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65829e68_acff_4d76_9fc0_3907abadd232.slice/crio-02c7242f56c98b26ed3c0e1b08fdfdecc38a8f5da32ac71a44610106de295603 WatchSource:0}: Error finding container 02c7242f56c98b26ed3c0e1b08fdfdecc38a8f5da32ac71a44610106de295603: Status 404 returned error can't find the container with id 02c7242f56c98b26ed3c0e1b08fdfdecc38a8f5da32ac71a44610106de295603 Jul 12 19:23:01.825983 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.825916 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c"] Jul 12 19:23:01.873192 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.873150 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j"] Jul 12 19:23:01.926254 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.926228 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-f6ng5"] Jul 12 19:23:01.943110 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.943076 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-f6ng5" event={"ID":"5e13f450-f8cd-43aa-9864-9670f84317ec","Type":"ContainerStarted","Data":"4fd8356b790842d17ac3c55333affb2abc3ef26808a729549c31a62ab138bd1c"} Jul 12 19:23:01.949308 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.949278 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" event={"ID":"65829e68-acff-4d76-9fc0-3907abadd232","Type":"ContainerStarted","Data":"02c7242f56c98b26ed3c0e1b08fdfdecc38a8f5da32ac71a44610106de295603"} Jul 12 19:23:01.955238 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.955142 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm4ws" event={"ID":"53fba168-256f-4c00-a11a-69b3a3224fff","Type":"ContainerStarted","Data":"f47188b84fd643e62c40575355fe86c5673df654d6fe7a6c73f6f0c680786440"} Jul 12 19:23:01.962606 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.961866 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" event={"ID":"928c8c62-9201-42ec-b710-560d624ec55c","Type":"ContainerStarted","Data":"171688af2b51825a4fa4998cd51ba2e1017ac84c92798e372608075596cd8601"} Jul 12 19:23:01.967065 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.967044 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ggg8r" event={"ID":"a8899441-0b5a-4714-9743-f03b4cc1ed86","Type":"ContainerStarted","Data":"92d1e8f41e721e7fe0b2683c42f17f4dcc1aa6d22dd17bda18945dd654e71e6f"} Jul 12 19:23:01.971918 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.971888 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" event={"ID":"7197c1de-7810-4eba-b11a-71b70e4c090d","Type":"ContainerStarted","Data":"977e0befd8ac986017092af91db7080c98c6b1324a35cda8926e79a87cecc52e"} Jul 12 19:23:01.977615 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.976755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" event={"ID":"05cec323-9469-437a-940d-1c370643767f","Type":"ContainerStarted","Data":"a7053861258c371b71597f18c09910e470315c6bae5fddce81414eb8c2529cba"} Jul 12 19:23:01.984724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:01.984673 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" podStartSLOduration=68.98465311 podStartE2EDuration="1m8.98465311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:01.982855834 +0000 UTC m=+70.665334294" watchObservedRunningTime="2026-07-12 19:23:01.98465311 +0000 UTC m=+70.667131573" Jul 12 19:23:02.343050 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:02.343013 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:02.346632 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:02.346382 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:02.990317 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:02.990215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ggg8r" event={"ID":"a8899441-0b5a-4714-9743-f03b4cc1ed86","Type":"ContainerStarted","Data":"714bca56e8e407fe63712bdd7e9d87081759f5d923bc1e7e8704d1815f0be78d"} Jul 12 19:23:02.990317 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:02.990275 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:02.992187 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:02.992145 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86d8bc5964-wgq5f" Jul 12 19:23:03.006872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:03.005142 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ggg8r" podStartSLOduration=69.472247151 podStartE2EDuration="1m11.005121288s" totalImagesPullingTime="1.532874137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:03.00481847 +0000 UTC m=+71.687296930" watchObservedRunningTime="2026-07-12 19:23:03.005121288 +0000 UTC m=+71.687599748" Jul 12 19:23:15.056368 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.056269 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" event={"ID":"7197c1de-7810-4eba-b11a-71b70e4c090d","Type":"ContainerStarted","Data":"2cbbe4354bfa12f3d021d1d3a494db8f9b91bd42a5d51ce7af56e772b8c27479"} Jul 12 19:23:15.058162 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.058100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" event={"ID":"05cec323-9469-437a-940d-1c370643767f","Type":"ContainerStarted","Data":"1f90558d6d53df09493997cd9da31b323014cffd5998a017da20baea505684fc"} Jul 12 19:23:15.059863 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.059836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmpsl" event={"ID":"566fd99f-bbc7-4532-83ef-f8cce366f737","Type":"ContainerStarted","Data":"34b0055dec62803228af7504faf00c1a353dbcb97b242dee772f2ad158b05263"} Jul 12 19:23:15.061921 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.061673 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" event={"ID":"df8f3888-df8e-480d-ae68-f6b16126ee83","Type":"ContainerStarted","Data":"713c8db8344789f88b7f174358e5b355fa1005a4d09756cbea23042fbf6ce8bc"} Jul 12 19:23:15.063427 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.063395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" event={"ID":"d3cfe81c-2ae4-48de-9292-cac871d9da5c","Type":"ContainerStarted","Data":"f313ae59be76feb042f03b193f2ac97d44a4bca686e84133991609650754625f"} Jul 12 19:23:15.064948 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.064925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" event={"ID":"65829e68-acff-4d76-9fc0-3907abadd232","Type":"ContainerStarted","Data":"3f321eb2216d6ab888435425f1e897f3c8ee0ccf1ce9e5a15fa7f1fcdbf8b860"} Jul 12 19:23:15.065375 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.065343 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:15.067419 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.067402 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" Jul 12 19:23:15.068120 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.067630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dm4ws" event={"ID":"53fba168-256f-4c00-a11a-69b3a3224fff","Type":"ContainerStarted","Data":"b0e2e958e546e888662619788cda846ec3dee3a8145d468ced1a90a4a56dc0ef"} Jul 12 19:23:15.069878 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.069838 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bblrg" event={"ID":"c57f6e17-8e69-4af8-9a1e-540b29c4874c","Type":"ContainerStarted","Data":"e8a3708c511630e5ff6fcd4afc304a0b9707e28aa5a7adb10664f921701ad45b"} Jul 12 19:23:15.073709 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.073564 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8586898f64-hrs8j" podStartSLOduration=1.989570826 podStartE2EDuration="14.073549236s" totalImagesPullingTime="12.08397841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:15.072329388 +0000 UTC m=+83.754807844" watchObservedRunningTime="2026-07-12 19:23:15.073549236 +0000 UTC m=+83.756027700" Jul 12 19:23:15.090805 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.090751 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dm4ws" podStartSLOduration=18.5883856 podStartE2EDuration="36.090732979s" totalImagesPullingTime="17.502347379s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:22:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:15.088952932 +0000 UTC m=+83.771431394" watchObservedRunningTime="2026-07-12 19:23:15.090732979 +0000 UTC m=+83.773211438" Jul 12 19:23:15.105577 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.105530 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb4668789-8wtk4" podStartSLOduration=1.926600672 podStartE2EDuration="14.105516118s" totalImagesPullingTime="12.178915446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:15.105298579 +0000 UTC m=+83.787777057" watchObservedRunningTime="2026-07-12 19:23:15.105516118 +0000 UTC m=+83.787994581" Jul 12 19:23:15.120133 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:15.120078 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bblrg" podStartSLOduration=69.185196386 podStartE2EDuration="1m23.120059607s" totalImagesPullingTime="13.934863221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 19:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:15.118822604 +0000 UTC m=+83.801301065" watchObservedRunningTime="2026-07-12 19:23:15.120059607 +0000 UTC m=+83.802538067" Jul 12 19:23:20.285066 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:20.285024 2565 patch_prober.go:28] interesting pod/image-registry-567dd678fc-ppt7c container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 12 19:23:20.285533 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:20.285089 2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" podUID="a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:23:21.982278 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:21.982242 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:23:22.003477 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.003379 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" podStartSLOduration=75.652884794 podStartE2EDuration="1m29.003353492s" totalImagesPullingTime="13.350468698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:15.133131221 +0000 UTC m=+83.815609681" watchObservedRunningTime="2026-07-12 19:23:22.003353492 +0000 UTC m=+90.685831955" Jul 12 19:23:22.107025 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.106984 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" event={"ID":"d3cfe81c-2ae4-48de-9292-cac871d9da5c","Type":"ContainerStarted","Data":"e58e4e28bbe6f789fad490b57594c5da8e55cd535e7d118a0d9018041f31a920"} Jul 12 19:23:22.109331 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.109302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-f6ng5" event={"ID":"5e13f450-f8cd-43aa-9864-9670f84317ec","Type":"ContainerStarted","Data":"ee742916580146e3e2bbd922f3c260cb09e0494edbe25f324436d91d43ca6b03"} Jul 12 19:23:22.111599 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.109848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-f6ng5" Jul 12 19:23:22.113149 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.113121 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" event={"ID":"7197c1de-7810-4eba-b11a-71b70e4c090d","Type":"ContainerStarted","Data":"e0b3c8bc7770841c9d1accedb0e0e5830460e159ee43e76ef981529985a6eac2"} Jul 12 19:23:22.113278 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.113165 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" event={"ID":"7197c1de-7810-4eba-b11a-71b70e4c090d","Type":"ContainerStarted","Data":"090fe81edacf7d97b4d236b5300931f5a82ddd383b8229ddda2afa279f319630"} Jul 12 19:23:22.115246 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.115218 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lmpsl" event={"ID":"566fd99f-bbc7-4532-83ef-f8cce366f737","Type":"ContainerStarted","Data":"c18953861d343cc74741e7d21cc15ebcdb1fed602bcf10a33c198d452e0a42cd"} Jul 12 19:23:22.115383 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.115359 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lmpsl" Jul 12 19:23:22.117734 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.117716 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-f6ng5" Jul 12 19:23:22.123501 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.123455 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-blrs8" podStartSLOduration=65.834787672 podStartE2EDuration="1m19.123441344s" totalImagesPullingTime="13.288653672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:22.121663395 +0000 UTC m=+90.804141857" watchObservedRunningTime="2026-07-12 19:23:22.123441344 +0000 UTC m=+90.805919804" Jul 12 19:23:22.139169 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.139116 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-f6ng5" podStartSLOduration=1.285960639 podStartE2EDuration="21.139099706s" totalImagesPullingTime="19.853139067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:22.137351912 +0000 UTC m=+90.819830435" watchObservedRunningTime="2026-07-12 19:23:22.139099706 +0000 UTC m=+90.821578167" Jul 12 19:23:22.154018 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.153972 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5896db89db-fnd9c" podStartSLOduration=9.018351966 podStartE2EDuration="21.15395591s" totalImagesPullingTime="12.135603944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:23:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:22.152822722 +0000 UTC m=+90.835301182" watchObservedRunningTime="2026-07-12 19:23:22.15395591 +0000 UTC m=+90.836434371" Jul 12 19:23:22.169929 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:22.169871 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lmpsl" podStartSLOduration=40.914373563 podStartE2EDuration="54.169854981s" totalImagesPullingTime="13.255481418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:22.168593062 +0000 UTC m=+90.851071525" watchObservedRunningTime="2026-07-12 19:23:22.169854981 +0000 UTC m=+90.852333442" Jul 12 19:23:23.465117 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:23.465079 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567dd678fc-ppt7c"] Jul 12 19:23:24.130776 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:24.130749 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:23:24.130958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:24.130793 2565 generic.go:356] "Generic (PLEG): container finished" podID="df8f3888-df8e-480d-ae68-f6b16126ee83" containerID="713c8db8344789f88b7f174358e5b355fa1005a4d09756cbea23042fbf6ce8bc" exitCode=2 Jul 12 19:23:24.130958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:24.130887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" event={"ID":"df8f3888-df8e-480d-ae68-f6b16126ee83","Type":"ContainerDied","Data":"713c8db8344789f88b7f174358e5b355fa1005a4d09756cbea23042fbf6ce8bc"} Jul 12 19:23:24.131284 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:24.131262 2565 scope.go:117] "RemoveContainer" containerID="713c8db8344789f88b7f174358e5b355fa1005a4d09756cbea23042fbf6ce8bc" Jul 12 19:23:25.138786 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:25.138752 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:23:25.139213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:25.138877 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-85cdb" event={"ID":"df8f3888-df8e-480d-ae68-f6b16126ee83","Type":"ContainerStarted","Data":"0544ff705388d8291f63d559f84f4f030ed86999fd29ddbcd273ce36bd7cbeda"} Jul 12 19:23:26.125647 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:26.125611 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lmpsl" Jul 12 19:23:27.886098 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.886066 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sd8c7"] Jul 12 19:23:27.920889 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.920862 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.923239 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.923218 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 12 19:23:27.923542 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.923413 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 12 19:23:27.923785 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.923768 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 12 19:23:27.924846 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.924132 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 12 19:23:27.924846 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.924358 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hchhq\"" Jul 12 19:23:27.990055 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990205 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990073 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbt6\" (UniqueName: \"kubernetes.io/projected/b0b133e1-6e87-423e-a535-745aa9beaeca-kube-api-access-xnbt6\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990205 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990157 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-root\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990323 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-wtmp\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990323 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990271 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-sys\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990323 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990293 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-textfile\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990468 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-accelerators-collector-config\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990468 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-tls\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:27.990468 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:27.990394 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0b133e1-6e87-423e-a535-745aa9beaeca-metrics-client-ca\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091056 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091018 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091237 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbt6\" (UniqueName: \"kubernetes.io/projected/b0b133e1-6e87-423e-a535-745aa9beaeca-kube-api-access-xnbt6\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091237 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-root\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091237 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091142 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-wtmp\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091237 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-sys\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091237 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091213 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-textfile\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091500 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-accelerators-collector-config\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091500 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091298 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-tls\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091500 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0b133e1-6e87-423e-a535-745aa9beaeca-metrics-client-ca\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.091648 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.091618 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-sys\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.092065 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.092039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0b133e1-6e87-423e-a535-745aa9beaeca-metrics-client-ca\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.092192 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.092048 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-root\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.092263 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.092231 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-wtmp\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.092337 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.092317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-textfile\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.092409 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:23:28.092335 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 12 19:23:28.092409 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:23:28.092390 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-tls podName:b0b133e1-6e87-423e-a535-745aa9beaeca nodeName:}" failed. No retries permitted until 2026-07-12 19:23:28.592371872 +0000 UTC m=+97.274850315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-tls") pod "node-exporter-sd8c7" (UID: "b0b133e1-6e87-423e-a535-745aa9beaeca") : secret "node-exporter-tls" not found Jul 12 19:23:28.092701 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.092668 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-accelerators-collector-config\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.094900 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.094874 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.102556 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.102537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbt6\" (UniqueName: \"kubernetes.io/projected/b0b133e1-6e87-423e-a535-745aa9beaeca-kube-api-access-xnbt6\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.595777 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.595735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-tls\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.598451 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.598421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b0b133e1-6e87-423e-a535-745aa9beaeca-node-exporter-tls\") pod \"node-exporter-sd8c7\" (UID: \"b0b133e1-6e87-423e-a535-745aa9beaeca\") " pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.834922 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:28.834891 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sd8c7" Jul 12 19:23:28.876725 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:23:28.876469 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b133e1_6e87_423e_a535_745aa9beaeca.slice/crio-e7f2b866d61e09a93595792eb38bfbe962a0fdfbf4998f86e09d3c31b0835c6d WatchSource:0}: Error finding container e7f2b866d61e09a93595792eb38bfbe962a0fdfbf4998f86e09d3c31b0835c6d: Status 404 returned error can't find the container with id e7f2b866d61e09a93595792eb38bfbe962a0fdfbf4998f86e09d3c31b0835c6d Jul 12 19:23:29.160238 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:29.160139 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd8c7" event={"ID":"b0b133e1-6e87-423e-a535-745aa9beaeca","Type":"ContainerStarted","Data":"e7f2b866d61e09a93595792eb38bfbe962a0fdfbf4998f86e09d3c31b0835c6d"} Jul 12 19:23:31.170293 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:31.170258 2565 generic.go:356] "Generic (PLEG): container finished" podID="b0b133e1-6e87-423e-a535-745aa9beaeca" containerID="a3ab6ac101f1610f1beff429baaf81a64e9c777c4856d84e7599ef757af3af52" exitCode=0 Jul 12 19:23:31.170681 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:31.170351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd8c7" event={"ID":"b0b133e1-6e87-423e-a535-745aa9beaeca","Type":"ContainerDied","Data":"a3ab6ac101f1610f1beff429baaf81a64e9c777c4856d84e7599ef757af3af52"} Jul 12 19:23:31.979916 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:31.979889 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kpjl6" Jul 12 19:23:32.176957 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:32.176912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd8c7" event={"ID":"b0b133e1-6e87-423e-a535-745aa9beaeca","Type":"ContainerStarted","Data":"d58387d0d04e5526acd5e7cb2416e886363b0d1e53c85ace9c72923d38a37ae5"} Jul 12 19:23:32.177429 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:32.177007 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sd8c7" event={"ID":"b0b133e1-6e87-423e-a535-745aa9beaeca","Type":"ContainerStarted","Data":"3e31c246d1e250f362ac982f7555f68af9a221dc96acfe975d18bb4dc6ff2481"} Jul 12 19:23:32.194219 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:32.194139 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sd8c7" podStartSLOduration=3.621372655 podStartE2EDuration="5.194118402s" totalImagesPullingTime="1.469442074s" totalInitContainerRuntime="103.303673ms" isStatefulPod=true podCreationTimestamp="2026-07-12 19:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:23:32.192307233 +0000 UTC m=+100.874785697" watchObservedRunningTime="2026-07-12 19:23:32.194118402 +0000 UTC m=+100.876596863" Jul 12 19:23:48.492315 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.492255 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" podUID="a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" containerName="registry" containerID="cri-o://fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802" gracePeriod=30 Jul 12 19:23:48.766104 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.766047 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:23:48.866829 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.866797 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-bound-sa-token\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867011 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.866840 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-ca-trust-extracted\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867011 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.866867 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-certificates\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867011 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.866918 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-image-registry-private-configuration\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867011 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.866946 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkn7c\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-kube-api-access-lkn7c\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867011 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.866973 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-trusted-ca\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867011 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.867001 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-installation-pull-secrets\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867354 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.867207 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") pod \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\" (UID: \"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8\") " Jul 12 19:23:48.867448 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.867421 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:23:48.867516 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.867454 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:23:48.867576 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.867544 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-certificates\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:48.867576 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.867565 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-trusted-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:48.869581 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.869530 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:23:48.869705 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.869628 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:23:48.869815 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.869795 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:23:48.869881 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.869864 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:23:48.870007 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.869992 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-kube-api-access-lkn7c" (OuterVolumeSpecName: "kube-api-access-lkn7c") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "kube-api-access-lkn7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:23:48.876162 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.876136 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" (UID: "a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 19:23:48.968193 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.968155 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-registry-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:48.968346 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.968208 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-bound-sa-token\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:48.968346 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.968217 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-ca-trust-extracted\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:48.968346 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.968229 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-image-registry-private-configuration\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:48.968346 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.968239 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkn7c\" (UniqueName: \"kubernetes.io/projected/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-kube-api-access-lkn7c\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:48.968346 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:48.968247 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8-installation-pull-secrets\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:23:49.244271 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.244236 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" containerID="fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802" exitCode=0 Jul 12 19:23:49.244457 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.244283 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" Jul 12 19:23:49.244457 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.244320 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" event={"ID":"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8","Type":"ContainerDied","Data":"fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802"} Jul 12 19:23:49.244457 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.244375 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-567dd678fc-ppt7c" event={"ID":"a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8","Type":"ContainerDied","Data":"90dfc4667791c3268a42170824f024ccd93862ca17ac26567ff0d797a189c695"} Jul 12 19:23:49.244457 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.244397 2565 scope.go:117] "RemoveContainer" containerID="fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802" Jul 12 19:23:49.245872 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.245850 2565 generic.go:356] "Generic (PLEG): container finished" podID="d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac" containerID="9d7115066223d979b00f0d2149ffc2b2444ed7a941bdb096186744301403ff9e" exitCode=0 Jul 12 19:23:49.245993 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.245883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" event={"ID":"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac","Type":"ContainerDied","Data":"9d7115066223d979b00f0d2149ffc2b2444ed7a941bdb096186744301403ff9e"} Jul 12 19:23:49.246216 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.246201 2565 scope.go:117] "RemoveContainer" containerID="9d7115066223d979b00f0d2149ffc2b2444ed7a941bdb096186744301403ff9e" Jul 12 19:23:49.253528 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.253510 2565 scope.go:117] "RemoveContainer" containerID="fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802" Jul 12 19:23:49.253824 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:23:49.253806 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802\": container with ID starting with fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802 not found: ID does not exist" containerID="fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802" Jul 12 19:23:49.253875 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.253833 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802"} err="failed to get container status \"fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802\": rpc error: code = NotFound desc = could not find container \"fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802\": container with ID starting with fc7c227d462dd643ccb8a7c1361ec4c11b77294b2bb05412b16c68cb099c1802 not found: ID does not exist" Jul 12 19:23:49.279525 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.279496 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-567dd678fc-ppt7c"] Jul 12 19:23:49.283821 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.283798 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-567dd678fc-ppt7c"] Jul 12 19:23:49.860549 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:49.860522 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" path="/var/lib/kubelet/pods/a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8/volumes" Jul 12 19:23:50.251517 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:23:50.251438 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-vjfb7" event={"ID":"d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac","Type":"ContainerStarted","Data":"4f4b4bb9b97466f07492e24b722a3459cd7c810f52e2dca76b22129ed92acb79"} Jul 12 19:24:10.333680 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:10.333646 2565 generic.go:356] "Generic (PLEG): container finished" podID="9e2e871f-6a10-44ae-a8d4-940e3c4e6489" containerID="63b91e803ad7c3a84cd3b5b63683f240360180a71b4b25200b782182f8d872f8" exitCode=0 Jul 12 19:24:10.334206 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:10.333717 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" event={"ID":"9e2e871f-6a10-44ae-a8d4-940e3c4e6489","Type":"ContainerDied","Data":"63b91e803ad7c3a84cd3b5b63683f240360180a71b4b25200b782182f8d872f8"} Jul 12 19:24:10.334206 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:10.334043 2565 scope.go:117] "RemoveContainer" containerID="63b91e803ad7c3a84cd3b5b63683f240360180a71b4b25200b782182f8d872f8" Jul 12 19:24:10.335301 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:10.335282 2565 generic.go:356] "Generic (PLEG): container finished" podID="6aa31e23-7983-4949-b4cc-4fe996a91a92" containerID="468bb710fcab24a19745ad33e33c30bc17edc012054d365d03353f5f7dd88c3c" exitCode=0 Jul 12 19:24:10.335386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:10.335319 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-52s44" event={"ID":"6aa31e23-7983-4949-b4cc-4fe996a91a92","Type":"ContainerDied","Data":"468bb710fcab24a19745ad33e33c30bc17edc012054d365d03353f5f7dd88c3c"} Jul 12 19:24:10.335590 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:10.335578 2565 scope.go:117] "RemoveContainer" containerID="468bb710fcab24a19745ad33e33c30bc17edc012054d365d03353f5f7dd88c3c" Jul 12 19:24:11.341096 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:11.341058 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-228q7" event={"ID":"9e2e871f-6a10-44ae-a8d4-940e3c4e6489","Type":"ContainerStarted","Data":"7f58546e1c7c2ee8e8d76c4cd33094bd338872c662297b262c3976b0fcea9213"} Jul 12 19:24:11.342830 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:24:11.342803 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-52s44" event={"ID":"6aa31e23-7983-4949-b4cc-4fe996a91a92","Type":"ContainerStarted","Data":"9320a4e447275392e6596501891b8a15afd33eb13cdcfb404f2a5357cb59b73f"} Jul 12 19:26:46.280103 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.280067 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd"] Jul 12 19:26:46.280571 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.280406 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" containerName="registry" Jul 12 19:26:46.280571 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.280418 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" containerName="registry" Jul 12 19:26:46.280571 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.280472 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2f4d1e2-ad06-4495-8b6b-0a6b70caefb8" containerName="registry" Jul 12 19:26:46.282271 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.282255 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.285171 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.285148 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 12 19:26:46.285323 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.285245 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 12 19:26:46.285323 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.285308 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4nbfd\"" Jul 12 19:26:46.285901 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.285882 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 12 19:26:46.286856 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.286843 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 12 19:26:46.307800 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.307769 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd"] Jul 12 19:26:46.465766 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.465736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/018d09cc-2ee0-436f-841c-cb4506fe05d1-manifests\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.465766 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.465770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/018d09cc-2ee0-436f-841c-cb4506fe05d1-tmp\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.465972 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.465790 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/018d09cc-2ee0-436f-841c-cb4506fe05d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.465972 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.465811 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/018d09cc-2ee0-436f-841c-cb4506fe05d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.465972 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.465828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6nd7\" (UniqueName: \"kubernetes.io/projected/018d09cc-2ee0-436f-841c-cb4506fe05d1-kube-api-access-l6nd7\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.567135 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.567048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/018d09cc-2ee0-436f-841c-cb4506fe05d1-manifests\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.567135 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.567085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/018d09cc-2ee0-436f-841c-cb4506fe05d1-tmp\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.567135 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.567106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/018d09cc-2ee0-436f-841c-cb4506fe05d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.567135 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.567137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/018d09cc-2ee0-436f-841c-cb4506fe05d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.567534 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.567158 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6nd7\" (UniqueName: \"kubernetes.io/projected/018d09cc-2ee0-436f-841c-cb4506fe05d1-kube-api-access-l6nd7\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.567534 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.567526 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/018d09cc-2ee0-436f-841c-cb4506fe05d1-manifests\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.569372 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.569343 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/018d09cc-2ee0-436f-841c-cb4506fe05d1-tmp\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.569565 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.569546 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/018d09cc-2ee0-436f-841c-cb4506fe05d1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.569610 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.569577 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/018d09cc-2ee0-436f-841c-cb4506fe05d1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.581650 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.581624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6nd7\" (UniqueName: \"kubernetes.io/projected/018d09cc-2ee0-436f-841c-cb4506fe05d1-kube-api-access-l6nd7\") pod \"opendatahub-operator-controller-manager-5449d7969f-zvhfd\" (UID: \"018d09cc-2ee0-436f-841c-cb4506fe05d1\") " pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.592504 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.592484 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:46.727630 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.727598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd"] Jul 12 19:26:46.730828 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:26:46.730806 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018d09cc_2ee0_436f_841c_cb4506fe05d1.slice/crio-771584beca0a21b5fe55c660d03e1dc1daa00701cc1a7e2c042ffc2bc7003c72 WatchSource:0}: Error finding container 771584beca0a21b5fe55c660d03e1dc1daa00701cc1a7e2c042ffc2bc7003c72: Status 404 returned error can't find the container with id 771584beca0a21b5fe55c660d03e1dc1daa00701cc1a7e2c042ffc2bc7003c72 Jul 12 19:26:46.934383 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:46.934337 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" event={"ID":"018d09cc-2ee0-436f-841c-cb4506fe05d1","Type":"ContainerStarted","Data":"771584beca0a21b5fe55c660d03e1dc1daa00701cc1a7e2c042ffc2bc7003c72"} Jul 12 19:26:49.951593 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:49.951557 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" event={"ID":"018d09cc-2ee0-436f-841c-cb4506fe05d1","Type":"ContainerStarted","Data":"c05b53ba36c8c0e76fe60ba61bfa4b6f5ce2e3abd4478c2854b0de0db304d4da"} Jul 12 19:26:50.956958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:50.956866 2565 generic.go:356] "Generic (PLEG): container finished" podID="018d09cc-2ee0-436f-841c-cb4506fe05d1" containerID="c05b53ba36c8c0e76fe60ba61bfa4b6f5ce2e3abd4478c2854b0de0db304d4da" exitCode=0 Jul 12 19:26:50.956958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:50.956941 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" event={"ID":"018d09cc-2ee0-436f-841c-cb4506fe05d1","Type":"ContainerDied","Data":"c05b53ba36c8c0e76fe60ba61bfa4b6f5ce2e3abd4478c2854b0de0db304d4da"} Jul 12 19:26:51.731099 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.731075 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:26:51.735253 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.735233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:26:51.737934 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.737915 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:26:51.742494 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.742472 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:26:51.748010 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.747985 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 12 19:26:51.962709 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.962671 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" event={"ID":"018d09cc-2ee0-436f-841c-cb4506fe05d1","Type":"ContainerStarted","Data":"9435a639e5eea5f1465dedfb6731368e3a4a572b2389151cfdd19d69a2f3a7b6"} Jul 12 19:26:51.963070 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.962796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:26:51.985015 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:26:51.984777 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" podStartSLOduration=1.811661039 podStartE2EDuration="5.984758634s" totalImagesPullingTime="3.103250223s" totalInitContainerRuntime="1.069847372s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:26:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:26:51.982096247 +0000 UTC m=+300.664574708" watchObservedRunningTime="2026-07-12 19:26:51.984758634 +0000 UTC m=+300.667237097" Jul 12 19:27:02.969771 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:02.969741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5449d7969f-zvhfd" Jul 12 19:27:17.562844 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.562813 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t6r72"] Jul 12 19:27:17.566208 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.566165 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:17.568721 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.568698 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 12 19:27:17.568816 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.568757 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z6v8z\"" Jul 12 19:27:17.576064 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.576042 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr"] Jul 12 19:27:17.579658 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.579635 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t6r72"] Jul 12 19:27:17.579769 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.579733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:17.581992 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.581973 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-2m4zf\"" Jul 12 19:27:17.582128 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.581992 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 12 19:27:17.589374 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.589349 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr"] Jul 12 19:27:17.629339 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.629306 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-cert\") pod \"kserve-controller-manager-d6f4b65b8-t6r72\" (UID: \"fb227f05-3565-4d27-b7f0-25a0a1ed13cd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:17.629499 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.629356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl52b\" (UniqueName: \"kubernetes.io/projected/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-kube-api-access-tl52b\") pod \"kserve-controller-manager-d6f4b65b8-t6r72\" (UID: \"fb227f05-3565-4d27-b7f0-25a0a1ed13cd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:17.629499 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.629430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53d57987-29a7-468b-8d48-fa747749f992-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-867fr\" (UID: \"53d57987-29a7-468b-8d48-fa747749f992\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:17.629580 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.629487 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cng26\" (UniqueName: \"kubernetes.io/projected/53d57987-29a7-468b-8d48-fa747749f992-kube-api-access-cng26\") pod \"llmisvc-controller-manager-777c6bd7d5-867fr\" (UID: \"53d57987-29a7-468b-8d48-fa747749f992\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:17.666147 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.666115 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw"] Jul 12 19:27:17.669264 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.669245 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.671699 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.671651 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 12 19:27:17.671699 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.671651 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 12 19:27:17.671865 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.671742 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mslwk\"" Jul 12 19:27:17.671865 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.671760 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 12 19:27:17.671977 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.671883 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 12 19:27:17.672200 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.672167 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 12 19:27:17.680152 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.680134 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw"] Jul 12 19:27:17.730231 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730169 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-cert\") pod \"kserve-controller-manager-d6f4b65b8-t6r72\" (UID: \"fb227f05-3565-4d27-b7f0-25a0a1ed13cd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:17.730231 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730232 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl52b\" (UniqueName: \"kubernetes.io/projected/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-kube-api-access-tl52b\") pod \"kserve-controller-manager-d6f4b65b8-t6r72\" (UID: \"fb227f05-3565-4d27-b7f0-25a0a1ed13cd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:17.730481 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e42822c4-13e0-4723-a43f-1e9be58e4e4b-manager-config\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.730481 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:27:17.730364 2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 12 19:27:17.730481 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730387 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e42822c4-13e0-4723-a43f-1e9be58e4e4b-metrics-cert\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.730481 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:27:17.730441 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-cert podName:fb227f05-3565-4d27-b7f0-25a0a1ed13cd nodeName:}" failed. No retries permitted until 2026-07-12 19:27:18.2304187 +0000 UTC m=+326.912897152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-cert") pod "kserve-controller-manager-d6f4b65b8-t6r72" (UID: "fb227f05-3565-4d27-b7f0-25a0a1ed13cd") : secret "kserve-webhook-server-cert" not found Jul 12 19:27:17.730701 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53d57987-29a7-468b-8d48-fa747749f992-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-867fr\" (UID: \"53d57987-29a7-468b-8d48-fa747749f992\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:17.730701 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cng26\" (UniqueName: \"kubernetes.io/projected/53d57987-29a7-468b-8d48-fa747749f992-kube-api-access-cng26\") pod \"llmisvc-controller-manager-777c6bd7d5-867fr\" (UID: \"53d57987-29a7-468b-8d48-fa747749f992\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:17.730701 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhsj\" (UniqueName: \"kubernetes.io/projected/e42822c4-13e0-4723-a43f-1e9be58e4e4b-kube-api-access-vdhsj\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.730701 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.730619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e42822c4-13e0-4723-a43f-1e9be58e4e4b-cert\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.730701 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:27:17.730631 2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 12 19:27:17.730701 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:27:17.730692 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53d57987-29a7-468b-8d48-fa747749f992-cert podName:53d57987-29a7-468b-8d48-fa747749f992 nodeName:}" failed. No retries permitted until 2026-07-12 19:27:18.230674621 +0000 UTC m=+326.913153061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/53d57987-29a7-468b-8d48-fa747749f992-cert") pod "llmisvc-controller-manager-777c6bd7d5-867fr" (UID: "53d57987-29a7-468b-8d48-fa747749f992") : secret "llmisvc-webhook-server-cert" not found Jul 12 19:27:17.742582 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.742544 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cng26\" (UniqueName: \"kubernetes.io/projected/53d57987-29a7-468b-8d48-fa747749f992-kube-api-access-cng26\") pod \"llmisvc-controller-manager-777c6bd7d5-867fr\" (UID: \"53d57987-29a7-468b-8d48-fa747749f992\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:17.742730 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.742544 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl52b\" (UniqueName: \"kubernetes.io/projected/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-kube-api-access-tl52b\") pod \"kserve-controller-manager-d6f4b65b8-t6r72\" (UID: \"fb227f05-3565-4d27-b7f0-25a0a1ed13cd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:17.832165 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.832065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhsj\" (UniqueName: \"kubernetes.io/projected/e42822c4-13e0-4723-a43f-1e9be58e4e4b-kube-api-access-vdhsj\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.832165 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.832126 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e42822c4-13e0-4723-a43f-1e9be58e4e4b-cert\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.832430 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.832214 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e42822c4-13e0-4723-a43f-1e9be58e4e4b-manager-config\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.832430 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.832244 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e42822c4-13e0-4723-a43f-1e9be58e4e4b-metrics-cert\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.832936 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.832917 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e42822c4-13e0-4723-a43f-1e9be58e4e4b-manager-config\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.835893 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.835861 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e42822c4-13e0-4723-a43f-1e9be58e4e4b-metrics-cert\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.835893 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.835882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e42822c4-13e0-4723-a43f-1e9be58e4e4b-cert\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.850204 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.850167 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhsj\" (UniqueName: \"kubernetes.io/projected/e42822c4-13e0-4723-a43f-1e9be58e4e4b-kube-api-access-vdhsj\") pod \"lws-controller-manager-79f8975998-rj5jw\" (UID: \"e42822c4-13e0-4723-a43f-1e9be58e4e4b\") " pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:17.981657 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:17.981617 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:18.115260 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.115226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw"] Jul 12 19:27:18.117881 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:27:18.117857 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode42822c4_13e0_4723_a43f_1e9be58e4e4b.slice/crio-b6d27ed22238d5b7d866283e683a99bd81d5819feb6c2e92feb504bcf29486eb WatchSource:0}: Error finding container b6d27ed22238d5b7d866283e683a99bd81d5819feb6c2e92feb504bcf29486eb: Status 404 returned error can't find the container with id b6d27ed22238d5b7d866283e683a99bd81d5819feb6c2e92feb504bcf29486eb Jul 12 19:27:18.119789 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.119769 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:27:18.234893 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.234857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53d57987-29a7-468b-8d48-fa747749f992-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-867fr\" (UID: \"53d57987-29a7-468b-8d48-fa747749f992\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:18.235053 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.234916 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-cert\") pod \"kserve-controller-manager-d6f4b65b8-t6r72\" (UID: \"fb227f05-3565-4d27-b7f0-25a0a1ed13cd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:18.237283 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.237254 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53d57987-29a7-468b-8d48-fa747749f992-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-867fr\" (UID: \"53d57987-29a7-468b-8d48-fa747749f992\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:18.237379 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.237294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fb227f05-3565-4d27-b7f0-25a0a1ed13cd-cert\") pod \"kserve-controller-manager-d6f4b65b8-t6r72\" (UID: \"fb227f05-3565-4d27-b7f0-25a0a1ed13cd\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:18.477078 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.476991 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:18.491288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.491259 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:18.619489 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.619110 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t6r72"] Jul 12 19:27:18.622765 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:27:18.622727 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb227f05_3565_4d27_b7f0_25a0a1ed13cd.slice/crio-5d075151f748ebd7c14a08f0f9ecc4697f23af75e746efbb961245144218b1bb WatchSource:0}: Error finding container 5d075151f748ebd7c14a08f0f9ecc4697f23af75e746efbb961245144218b1bb: Status 404 returned error can't find the container with id 5d075151f748ebd7c14a08f0f9ecc4697f23af75e746efbb961245144218b1bb Jul 12 19:27:18.652762 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:18.652735 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr"] Jul 12 19:27:18.656128 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:27:18.656101 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod53d57987_29a7_468b_8d48_fa747749f992.slice/crio-1b504104201a1d41cd63398a9d5fc4aa75bf3c52d8ecb45d4c675e930fb27de0 WatchSource:0}: Error finding container 1b504104201a1d41cd63398a9d5fc4aa75bf3c52d8ecb45d4c675e930fb27de0: Status 404 returned error can't find the container with id 1b504104201a1d41cd63398a9d5fc4aa75bf3c52d8ecb45d4c675e930fb27de0 Jul 12 19:27:19.078601 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:19.078558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" event={"ID":"e42822c4-13e0-4723-a43f-1e9be58e4e4b","Type":"ContainerStarted","Data":"b6d27ed22238d5b7d866283e683a99bd81d5819feb6c2e92feb504bcf29486eb"} Jul 12 19:27:19.080010 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:19.079957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" event={"ID":"53d57987-29a7-468b-8d48-fa747749f992","Type":"ContainerStarted","Data":"1b504104201a1d41cd63398a9d5fc4aa75bf3c52d8ecb45d4c675e930fb27de0"} Jul 12 19:27:19.081457 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:19.081405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" event={"ID":"fb227f05-3565-4d27-b7f0-25a0a1ed13cd","Type":"ContainerStarted","Data":"5d075151f748ebd7c14a08f0f9ecc4697f23af75e746efbb961245144218b1bb"} Jul 12 19:27:23.102758 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.102718 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" event={"ID":"e42822c4-13e0-4723-a43f-1e9be58e4e4b","Type":"ContainerStarted","Data":"534532d4df7d944003f4e67906b21c0e08d47faaee492f5d393a135f013357c5"} Jul 12 19:27:23.103239 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.102835 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:23.104391 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.104365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" event={"ID":"53d57987-29a7-468b-8d48-fa747749f992","Type":"ContainerStarted","Data":"b2acdda84e397006e19a04ee6cb22ddb72f0f7458361df1c12e4756a73d78f46"} Jul 12 19:27:23.104513 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.104489 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:23.105833 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.105810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" event={"ID":"fb227f05-3565-4d27-b7f0-25a0a1ed13cd","Type":"ContainerStarted","Data":"e9704855796f437a688ee22604314e256eda2e1c51d6db066f8d29c83bd61a15"} Jul 12 19:27:23.105952 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.105936 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:27:23.119594 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.119552 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" podStartSLOduration=2.670417133 podStartE2EDuration="6.119538486s" totalImagesPullingTime="3.449121353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:23.118551689 +0000 UTC m=+331.801030149" watchObservedRunningTime="2026-07-12 19:27:23.119538486 +0000 UTC m=+331.802016924" Jul 12 19:27:23.134172 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.134123 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" podStartSLOduration=2.294983892 podStartE2EDuration="6.134108756s" totalImagesPullingTime="3.839124864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:23.132548181 +0000 UTC m=+331.815026640" watchObservedRunningTime="2026-07-12 19:27:23.134108756 +0000 UTC m=+331.816587216" Jul 12 19:27:23.146809 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:23.146758 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" podStartSLOduration=3.118357193 podStartE2EDuration="6.146742988s" totalImagesPullingTime="3.028385795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:27:23.146385658 +0000 UTC m=+331.828864110" watchObservedRunningTime="2026-07-12 19:27:23.146742988 +0000 UTC m=+331.829221449" Jul 12 19:27:34.113424 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:34.113391 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-79f8975998-rj5jw" Jul 12 19:27:54.112433 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:54.112355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-867fr" Jul 12 19:27:54.115206 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:27:54.115165 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t6r72" Jul 12 19:29:49.521117 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.521087 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:29:49.524490 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.524470 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.526804 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.526784 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 12 19:29:49.526909 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.526821 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 12 19:29:49.526909 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.526885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sfq5m\"" Jul 12 19:29:49.527676 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.527662 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 12 19:29:49.531805 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.531782 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:29:49.558412 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.558391 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:29:49.605887 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.605858 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed027eec-553f-4c48-b37d-023b29c4be1c-config-file\") pod \"limitador-limitador-54d6d988d5-qcz8n\" (UID: \"ed027eec-553f-4c48-b37d-023b29c4be1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.606009 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.605928 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj6d8\" (UniqueName: \"kubernetes.io/projected/ed027eec-553f-4c48-b37d-023b29c4be1c-kube-api-access-tj6d8\") pod \"limitador-limitador-54d6d988d5-qcz8n\" (UID: \"ed027eec-553f-4c48-b37d-023b29c4be1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.706961 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.706931 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj6d8\" (UniqueName: \"kubernetes.io/projected/ed027eec-553f-4c48-b37d-023b29c4be1c-kube-api-access-tj6d8\") pod \"limitador-limitador-54d6d988d5-qcz8n\" (UID: \"ed027eec-553f-4c48-b37d-023b29c4be1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.707094 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.706970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed027eec-553f-4c48-b37d-023b29c4be1c-config-file\") pod \"limitador-limitador-54d6d988d5-qcz8n\" (UID: \"ed027eec-553f-4c48-b37d-023b29c4be1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.707615 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.707596 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed027eec-553f-4c48-b37d-023b29c4be1c-config-file\") pod \"limitador-limitador-54d6d988d5-qcz8n\" (UID: \"ed027eec-553f-4c48-b37d-023b29c4be1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.713898 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.713876 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj6d8\" (UniqueName: \"kubernetes.io/projected/ed027eec-553f-4c48-b37d-023b29c4be1c-kube-api-access-tj6d8\") pod \"limitador-limitador-54d6d988d5-qcz8n\" (UID: \"ed027eec-553f-4c48-b37d-023b29c4be1c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.836385 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.836307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:49.969097 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:49.969069 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:29:49.971388 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:29:49.971362 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded027eec_553f_4c48_b37d_023b29c4be1c.slice/crio-7925d01b2fcc29b393102b464bb1e29a05a2b8421b4e9319683a95712ce86235 WatchSource:0}: Error finding container 7925d01b2fcc29b393102b464bb1e29a05a2b8421b4e9319683a95712ce86235: Status 404 returned error can't find the container with id 7925d01b2fcc29b393102b464bb1e29a05a2b8421b4e9319683a95712ce86235 Jul 12 19:29:50.750814 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:50.750704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" event={"ID":"ed027eec-553f-4c48-b37d-023b29c4be1c","Type":"ContainerStarted","Data":"7925d01b2fcc29b393102b464bb1e29a05a2b8421b4e9319683a95712ce86235"} Jul 12 19:29:52.761671 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:52.761580 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" event={"ID":"ed027eec-553f-4c48-b37d-023b29c4be1c","Type":"ContainerStarted","Data":"eb89dcd93c7fc6aea6536c3495cf4087584197e2c8f8b6150e0eec48012b2739"} Jul 12 19:29:52.762004 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:52.761778 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:29:52.776500 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:29:52.776453 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" podStartSLOduration=1.24289659 podStartE2EDuration="3.776438649s" totalImagesPullingTime="2.533542059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:29:52.775155261 +0000 UTC m=+481.457633720" watchObservedRunningTime="2026-07-12 19:29:52.776438649 +0000 UTC m=+481.458917108" Jul 12 19:30:03.767134 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:03.767103 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-qcz8n" Jul 12 19:30:47.872973 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.872897 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:30:47.876243 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.876227 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 12 19:30:47.878593 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.878570 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\"" Jul 12 19:30:47.878593 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.878584 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jul 12 19:30:47.878753 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.878570 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\"" Jul 12 19:30:47.879357 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.879340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"default-dockercfg-ghrkl\"" Jul 12 19:30:47.884791 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.884765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:30:47.961697 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:47.961661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpqbn\" (UniqueName: \"kubernetes.io/projected/45f8b4ba-5b82-4854-bf6d-4f490dfb28e6-kube-api-access-kpqbn\") pod \"maas-keycloak-0\" (UID: \"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:30:48.063084 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:48.063057 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpqbn\" (UniqueName: \"kubernetes.io/projected/45f8b4ba-5b82-4854-bf6d-4f490dfb28e6-kube-api-access-kpqbn\") pod \"maas-keycloak-0\" (UID: \"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:30:48.070492 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:48.070464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpqbn\" (UniqueName: \"kubernetes.io/projected/45f8b4ba-5b82-4854-bf6d-4f490dfb28e6-kube-api-access-kpqbn\") pod \"maas-keycloak-0\" (UID: \"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:30:48.186592 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:48.186508 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 12 19:30:48.314349 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:48.314327 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:30:48.316583 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:30:48.316560 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f8b4ba_5b82_4854_bf6d_4f490dfb28e6.slice/crio-d933da13eb6177cdd4dc29fb9b89001719f5bd0410400d79962f600f8484175d WatchSource:0}: Error finding container d933da13eb6177cdd4dc29fb9b89001719f5bd0410400d79962f600f8484175d: Status 404 returned error can't find the container with id d933da13eb6177cdd4dc29fb9b89001719f5bd0410400d79962f600f8484175d Jul 12 19:30:49.010750 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:49.010688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6","Type":"ContainerStarted","Data":"d933da13eb6177cdd4dc29fb9b89001719f5bd0410400d79962f600f8484175d"} Jul 12 19:30:53.184055 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:53.184029 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jul 12 19:30:54.049631 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:54.049585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6","Type":"ContainerStarted","Data":"c2648079a28156722e5ca2a4323e70ba818f9d7fa5b7b7cf5685c6f639f3ac60"} Jul 12 19:30:54.068221 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:54.068155 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="keycloak-system/maas-keycloak-0" podStartSLOduration=2.204339404 podStartE2EDuration="7.068139618s" totalImagesPullingTime="4.863800214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:30:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:30:54.06472899 +0000 UTC m=+542.747207451" watchObservedRunningTime="2026-07-12 19:30:54.068139618 +0000 UTC m=+542.750618078" Jul 12 19:30:54.187721 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:54.187679 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="keycloak-system/maas-keycloak-0" Jul 12 19:30:54.189668 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:54.189632 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:30:55.187774 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:55.187715 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:30:56.187906 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:56.187859 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:30:57.187747 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:57.187696 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:30:58.187435 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:58.187380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="keycloak-system/maas-keycloak-0" Jul 12 19:30:58.188055 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:58.187771 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:30:59.187419 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:30:59.187374 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:00.188075 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:00.188022 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:01.187478 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:01.187420 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:02.187291 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:02.187237 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:03.187812 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:03.187756 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:04.187846 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:04.187808 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:05.187035 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:05.186986 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:06.187298 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:06.187244 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.31:9000/health/started\": dial tcp 10.132.0.31:9000: connect: connection refused" Jul 12 19:31:07.293279 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:07.293238 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="keycloak-system/maas-keycloak-0" Jul 12 19:31:07.311296 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:07.311254 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:31:17.300072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:17.300035 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="keycloak-system/maas-keycloak-0" Jul 12 19:31:24.245579 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.245542 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cff46df46-2hbzh"] Jul 12 19:31:24.249336 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.249312 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.252342 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.252318 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 12 19:31:24.252429 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.252327 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 12 19:31:24.252429 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.252331 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sr8w4\"" Jul 12 19:31:24.259346 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.259324 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cff46df46-2hbzh"] Jul 12 19:31:24.396801 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.396764 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8szv\" (UniqueName: \"kubernetes.io/projected/e969a65f-4d31-460f-bf92-09f448666ea4-kube-api-access-l8szv\") pod \"maas-controller-cff46df46-2hbzh\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.396801 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.396805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e969a65f-4d31-460f-bf92-09f448666ea4-webhook-cert\") pod \"maas-controller-cff46df46-2hbzh\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.409042 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.409011 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-557cf6647c-vlzkq"] Jul 12 19:31:24.412909 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.412889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.421679 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.421654 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557cf6647c-vlzkq"] Jul 12 19:31:24.497459 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.497393 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8szv\" (UniqueName: \"kubernetes.io/projected/e969a65f-4d31-460f-bf92-09f448666ea4-kube-api-access-l8szv\") pod \"maas-controller-cff46df46-2hbzh\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.497459 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.497433 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e969a65f-4d31-460f-bf92-09f448666ea4-webhook-cert\") pod \"maas-controller-cff46df46-2hbzh\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.499902 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.499881 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e969a65f-4d31-460f-bf92-09f448666ea4-webhook-cert\") pod \"maas-controller-cff46df46-2hbzh\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.504965 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.504941 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8szv\" (UniqueName: \"kubernetes.io/projected/e969a65f-4d31-460f-bf92-09f448666ea4-kube-api-access-l8szv\") pod \"maas-controller-cff46df46-2hbzh\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.560397 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.560375 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:24.598707 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.598673 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/886ddc55-627b-4578-af55-aa55c9ad2eb8-webhook-cert\") pod \"maas-controller-557cf6647c-vlzkq\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.598971 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.598940 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzbr\" (UniqueName: \"kubernetes.io/projected/886ddc55-627b-4578-af55-aa55c9ad2eb8-kube-api-access-zbzbr\") pod \"maas-controller-557cf6647c-vlzkq\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.685519 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.685491 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cff46df46-2hbzh"] Jul 12 19:31:24.686999 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:31:24.686975 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode969a65f_4d31_460f_bf92_09f448666ea4.slice/crio-c1974831390fd6292786b28ca9c835634dd79943217bc68c579adbacd9924c8d WatchSource:0}: Error finding container c1974831390fd6292786b28ca9c835634dd79943217bc68c579adbacd9924c8d: Status 404 returned error can't find the container with id c1974831390fd6292786b28ca9c835634dd79943217bc68c579adbacd9924c8d Jul 12 19:31:24.700284 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.700263 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/886ddc55-627b-4578-af55-aa55c9ad2eb8-webhook-cert\") pod \"maas-controller-557cf6647c-vlzkq\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.700406 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.700310 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzbr\" (UniqueName: \"kubernetes.io/projected/886ddc55-627b-4578-af55-aa55c9ad2eb8-kube-api-access-zbzbr\") pod \"maas-controller-557cf6647c-vlzkq\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.702711 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.702689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/886ddc55-627b-4578-af55-aa55c9ad2eb8-webhook-cert\") pod \"maas-controller-557cf6647c-vlzkq\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.707682 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.707658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzbr\" (UniqueName: \"kubernetes.io/projected/886ddc55-627b-4578-af55-aa55c9ad2eb8-kube-api-access-zbzbr\") pod \"maas-controller-557cf6647c-vlzkq\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.723472 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.723452 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:24.847663 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:24.847631 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557cf6647c-vlzkq"] Jul 12 19:31:24.849293 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:31:24.849267 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886ddc55_627b_4578_af55_aa55c9ad2eb8.slice/crio-1bb3d437899afc9cf8dcb1d71ec8a83a419660d46cf48c1ce820a9eff9fd6c7c WatchSource:0}: Error finding container 1bb3d437899afc9cf8dcb1d71ec8a83a419660d46cf48c1ce820a9eff9fd6c7c: Status 404 returned error can't find the container with id 1bb3d437899afc9cf8dcb1d71ec8a83a419660d46cf48c1ce820a9eff9fd6c7c Jul 12 19:31:25.210244 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:25.210213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cff46df46-2hbzh" event={"ID":"e969a65f-4d31-460f-bf92-09f448666ea4","Type":"ContainerStarted","Data":"c1974831390fd6292786b28ca9c835634dd79943217bc68c579adbacd9924c8d"} Jul 12 19:31:25.211277 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:25.211252 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557cf6647c-vlzkq" event={"ID":"886ddc55-627b-4578-af55-aa55c9ad2eb8","Type":"ContainerStarted","Data":"1bb3d437899afc9cf8dcb1d71ec8a83a419660d46cf48c1ce820a9eff9fd6c7c"} Jul 12 19:31:28.233627 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:28.233582 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cff46df46-2hbzh" event={"ID":"e969a65f-4d31-460f-bf92-09f448666ea4","Type":"ContainerStarted","Data":"7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566"} Jul 12 19:31:28.234080 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:28.233685 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:28.235088 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:28.235064 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557cf6647c-vlzkq" event={"ID":"886ddc55-627b-4578-af55-aa55c9ad2eb8","Type":"ContainerStarted","Data":"32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc"} Jul 12 19:31:28.235292 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:28.235116 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:28.275009 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:28.274968 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cff46df46-2hbzh" podStartSLOduration=1.452479525 podStartE2EDuration="4.274956084s" totalImagesPullingTime="2.822476559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:28.273385173 +0000 UTC m=+576.955863633" watchObservedRunningTime="2026-07-12 19:31:28.274956084 +0000 UTC m=+576.957434543" Jul 12 19:31:28.309224 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:28.309172 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-557cf6647c-vlzkq" podStartSLOduration=1.637298988 podStartE2EDuration="4.309158041s" totalImagesPullingTime="2.671859053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:31:28.30833042 +0000 UTC m=+576.990808873" watchObservedRunningTime="2026-07-12 19:31:28.309158041 +0000 UTC m=+576.991636538" Jul 12 19:31:39.243928 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.243891 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:31:39.244386 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.243969 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:39.300883 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.300849 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cff46df46-2hbzh"] Jul 12 19:31:39.301092 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.301058 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cff46df46-2hbzh" podUID="e969a65f-4d31-460f-bf92-09f448666ea4" containerName="manager" containerID="cri-o://7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566" gracePeriod=30 Jul 12 19:31:39.553279 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.553257 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:39.618405 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.618375 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e969a65f-4d31-460f-bf92-09f448666ea4-webhook-cert\") pod \"e969a65f-4d31-460f-bf92-09f448666ea4\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " Jul 12 19:31:39.618517 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.618433 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8szv\" (UniqueName: \"kubernetes.io/projected/e969a65f-4d31-460f-bf92-09f448666ea4-kube-api-access-l8szv\") pod \"e969a65f-4d31-460f-bf92-09f448666ea4\" (UID: \"e969a65f-4d31-460f-bf92-09f448666ea4\") " Jul 12 19:31:39.620521 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.620499 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e969a65f-4d31-460f-bf92-09f448666ea4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e969a65f-4d31-460f-bf92-09f448666ea4" (UID: "e969a65f-4d31-460f-bf92-09f448666ea4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:31:39.620521 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.620505 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e969a65f-4d31-460f-bf92-09f448666ea4-kube-api-access-l8szv" (OuterVolumeSpecName: "kube-api-access-l8szv") pod "e969a65f-4d31-460f-bf92-09f448666ea4" (UID: "e969a65f-4d31-460f-bf92-09f448666ea4"). InnerVolumeSpecName "kube-api-access-l8szv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:31:39.719315 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.719287 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8szv\" (UniqueName: \"kubernetes.io/projected/e969a65f-4d31-460f-bf92-09f448666ea4-kube-api-access-l8szv\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:31:39.719315 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:39.719314 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e969a65f-4d31-460f-bf92-09f448666ea4-webhook-cert\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:31:40.287931 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.287848 2565 generic.go:356] "Generic (PLEG): container finished" podID="e969a65f-4d31-460f-bf92-09f448666ea4" containerID="7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566" exitCode=0 Jul 12 19:31:40.287931 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.287902 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cff46df46-2hbzh" Jul 12 19:31:40.288393 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.287935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cff46df46-2hbzh" event={"ID":"e969a65f-4d31-460f-bf92-09f448666ea4","Type":"ContainerDied","Data":"7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566"} Jul 12 19:31:40.288393 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.288011 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cff46df46-2hbzh" event={"ID":"e969a65f-4d31-460f-bf92-09f448666ea4","Type":"ContainerDied","Data":"c1974831390fd6292786b28ca9c835634dd79943217bc68c579adbacd9924c8d"} Jul 12 19:31:40.288393 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.288033 2565 scope.go:117] "RemoveContainer" containerID="7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566" Jul 12 19:31:40.296352 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.296338 2565 scope.go:117] "RemoveContainer" containerID="7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566" Jul 12 19:31:40.296595 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:31:40.296577 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566\": container with ID starting with 7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566 not found: ID does not exist" containerID="7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566" Jul 12 19:31:40.296636 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.296602 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566"} err="failed to get container status \"7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566\": rpc error: code = NotFound desc = could not find container \"7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566\": container with ID starting with 7e1221697f2028c594b3f71df10c552a36215c6e5aec5b7a9289d4d5c2b55566 not found: ID does not exist" Jul 12 19:31:40.303719 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.303698 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cff46df46-2hbzh"] Jul 12 19:31:40.305807 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:40.305789 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cff46df46-2hbzh"] Jul 12 19:31:41.861243 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:41.861211 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e969a65f-4d31-460f-bf92-09f448666ea4" path="/var/lib/kubelet/pods/e969a65f-4d31-460f-bf92-09f448666ea4/volumes" Jul 12 19:31:51.762717 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:51.762689 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:31:51.765958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:51.765933 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:31:51.766582 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:51.766552 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:31:51.769671 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:51.769651 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:31:57.600313 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.600268 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-57bdd766bb-cxs9w"] Jul 12 19:31:57.602765 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.600769 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e969a65f-4d31-460f-bf92-09f448666ea4" containerName="manager" Jul 12 19:31:57.602765 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.600785 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e969a65f-4d31-460f-bf92-09f448666ea4" containerName="manager" Jul 12 19:31:57.602765 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.600861 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e969a65f-4d31-460f-bf92-09f448666ea4" containerName="manager" Jul 12 19:31:57.603652 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.603634 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.605769 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.605747 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 12 19:31:57.605860 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.605840 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gfkkn\"" Jul 12 19:31:57.613520 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.613497 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-57bdd766bb-cxs9w"] Jul 12 19:31:57.657307 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.657283 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzz6\" (UniqueName: \"kubernetes.io/projected/c84c2eb5-2a76-432c-8ffd-df78e511e186-kube-api-access-8wzz6\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.657419 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.657315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84c2eb5-2a76-432c-8ffd-df78e511e186-openshift-service-ca\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.657419 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.657352 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84c2eb5-2a76-432c-8ffd-df78e511e186-maas-api-tls\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.758610 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.758580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzz6\" (UniqueName: \"kubernetes.io/projected/c84c2eb5-2a76-432c-8ffd-df78e511e186-kube-api-access-8wzz6\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.758610 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.758613 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84c2eb5-2a76-432c-8ffd-df78e511e186-openshift-service-ca\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.758810 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.758649 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84c2eb5-2a76-432c-8ffd-df78e511e186-maas-api-tls\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.759363 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.759344 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c84c2eb5-2a76-432c-8ffd-df78e511e186-openshift-service-ca\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.761147 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.761126 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c84c2eb5-2a76-432c-8ffd-df78e511e186-maas-api-tls\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.766212 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.766191 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzz6\" (UniqueName: \"kubernetes.io/projected/c84c2eb5-2a76-432c-8ffd-df78e511e186-kube-api-access-8wzz6\") pod \"maas-api-57bdd766bb-cxs9w\" (UID: \"c84c2eb5-2a76-432c-8ffd-df78e511e186\") " pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:57.915515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:57.915436 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:31:58.050509 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:58.050480 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-57bdd766bb-cxs9w"] Jul 12 19:31:58.052868 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:58.052846 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:31:58.054403 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:31:58.054382 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc84c2eb5_2a76_432c_8ffd_df78e511e186.slice/crio-e39edfc82c5c20f9ac3e31f421f57be368d628d867e5c8a1e5d35ef2aa87e7a7 WatchSource:0}: Error finding container e39edfc82c5c20f9ac3e31f421f57be368d628d867e5c8a1e5d35ef2aa87e7a7: Status 404 returned error can't find the container with id e39edfc82c5c20f9ac3e31f421f57be368d628d867e5c8a1e5d35ef2aa87e7a7 Jul 12 19:31:58.369400 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:31:58.369365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-57bdd766bb-cxs9w" event={"ID":"c84c2eb5-2a76-432c-8ffd-df78e511e186","Type":"ContainerStarted","Data":"e39edfc82c5c20f9ac3e31f421f57be368d628d867e5c8a1e5d35ef2aa87e7a7"} Jul 12 19:32:00.381408 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:00.381365 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-57bdd766bb-cxs9w" event={"ID":"c84c2eb5-2a76-432c-8ffd-df78e511e186","Type":"ContainerStarted","Data":"11ee7051f0e9cfdeffdb6bea356d4a734fa75e595210c379071ac3833e8a9784"} Jul 12 19:32:00.382007 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:00.381429 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:32:00.397565 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:00.397523 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-57bdd766bb-cxs9w" podStartSLOduration=2.023142044 podStartE2EDuration="3.397509506s" totalImagesPullingTime="1.374367462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:31:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:32:00.396323046 +0000 UTC m=+609.078801506" watchObservedRunningTime="2026-07-12 19:32:00.397509506 +0000 UTC m=+609.079987966" Jul 12 19:32:06.391028 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:06.390997 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-57bdd766bb-cxs9w" Jul 12 19:32:07.777289 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:07.777258 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:32:07.777682 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:07.777451 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="keycloak-system/maas-keycloak-0" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" containerID="cri-o://c2648079a28156722e5ca2a4323e70ba818f9d7fa5b7b7cf5685c6f639f3ac60" gracePeriod=30 Jul 12 19:32:09.422674 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:09.422642 2565 generic.go:356] "Generic (PLEG): container finished" podID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerID="c2648079a28156722e5ca2a4323e70ba818f9d7fa5b7b7cf5685c6f639f3ac60" exitCode=143 Jul 12 19:32:09.423073 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:09.422712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6","Type":"ContainerDied","Data":"c2648079a28156722e5ca2a4323e70ba818f9d7fa5b7b7cf5685c6f639f3ac60"} Jul 12 19:32:09.828664 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:09.828644 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:09.964997 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:09.964920 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpqbn\" (UniqueName: \"kubernetes.io/projected/45f8b4ba-5b82-4854-bf6d-4f490dfb28e6-kube-api-access-kpqbn\") pod \"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6\" (UID: \"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6\") " Jul 12 19:32:09.967084 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:09.967059 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f8b4ba-5b82-4854-bf6d-4f490dfb28e6-kube-api-access-kpqbn" (OuterVolumeSpecName: "kube-api-access-kpqbn") pod "45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" (UID: "45f8b4ba-5b82-4854-bf6d-4f490dfb28e6"). InnerVolumeSpecName "kube-api-access-kpqbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:32:10.066110 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.066070 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpqbn\" (UniqueName: \"kubernetes.io/projected/45f8b4ba-5b82-4854-bf6d-4f490dfb28e6-kube-api-access-kpqbn\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:32:10.428010 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.427978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"45f8b4ba-5b82-4854-bf6d-4f490dfb28e6","Type":"ContainerDied","Data":"d933da13eb6177cdd4dc29fb9b89001719f5bd0410400d79962f600f8484175d"} Jul 12 19:32:10.428449 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.428029 2565 scope.go:117] "RemoveContainer" containerID="c2648079a28156722e5ca2a4323e70ba818f9d7fa5b7b7cf5685c6f639f3ac60" Jul 12 19:32:10.428449 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.427994 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.447428 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.447406 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:32:10.450931 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.450909 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:32:10.471551 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.471532 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:32:10.471851 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.471839 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" Jul 12 19:32:10.471892 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.471853 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" Jul 12 19:32:10.471928 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.471914 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" containerName="keycloak" Jul 12 19:32:10.475796 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.475768 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.478091 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.478066 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"keycloak-test-realms\"" Jul 12 19:32:10.478216 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.478095 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"maas-keycloak-initial-admin\"" Jul 12 19:32:10.478216 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.478113 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\"" Jul 12 19:32:10.478216 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.478131 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\"" Jul 12 19:32:10.478216 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.478095 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"default-dockercfg-ghrkl\"" Jul 12 19:32:10.482866 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.482846 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:32:10.570148 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.570110 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/4e890b52-05a8-494b-946e-8ad3db3c048a-test-realms\") pod \"maas-keycloak-0\" (UID: \"4e890b52-05a8-494b-946e-8ad3db3c048a\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.570148 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.570151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2gb\" (UniqueName: \"kubernetes.io/projected/4e890b52-05a8-494b-946e-8ad3db3c048a-kube-api-access-7t2gb\") pod \"maas-keycloak-0\" (UID: \"4e890b52-05a8-494b-946e-8ad3db3c048a\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.671033 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.671003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/4e890b52-05a8-494b-946e-8ad3db3c048a-test-realms\") pod \"maas-keycloak-0\" (UID: \"4e890b52-05a8-494b-946e-8ad3db3c048a\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.671209 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.671038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2gb\" (UniqueName: \"kubernetes.io/projected/4e890b52-05a8-494b-946e-8ad3db3c048a-kube-api-access-7t2gb\") pod \"maas-keycloak-0\" (UID: \"4e890b52-05a8-494b-946e-8ad3db3c048a\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.671696 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.671677 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"test-realms\" (UniqueName: \"kubernetes.io/configmap/4e890b52-05a8-494b-946e-8ad3db3c048a-test-realms\") pod \"maas-keycloak-0\" (UID: \"4e890b52-05a8-494b-946e-8ad3db3c048a\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.678831 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.678766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2gb\" (UniqueName: \"kubernetes.io/projected/4e890b52-05a8-494b-946e-8ad3db3c048a-kube-api-access-7t2gb\") pod \"maas-keycloak-0\" (UID: \"4e890b52-05a8-494b-946e-8ad3db3c048a\") " pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:10.785335 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:10.785304 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:11.114723 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:11.114698 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/maas-keycloak-0"] Jul 12 19:32:11.117246 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:32:11.117209 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e890b52_05a8_494b_946e_8ad3db3c048a.slice/crio-c73cac9ba345863f339318f4d9917048d47fda9d7c014f14105d817074144f64 WatchSource:0}: Error finding container c73cac9ba345863f339318f4d9917048d47fda9d7c014f14105d817074144f64: Status 404 returned error can't find the container with id c73cac9ba345863f339318f4d9917048d47fda9d7c014f14105d817074144f64 Jul 12 19:32:11.434082 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:11.434047 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"4e890b52-05a8-494b-946e-8ad3db3c048a","Type":"ContainerStarted","Data":"c73cac9ba345863f339318f4d9917048d47fda9d7c014f14105d817074144f64"} Jul 12 19:32:11.863853 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:11.863818 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f8b4ba-5b82-4854-bf6d-4f490dfb28e6" path="/var/lib/kubelet/pods/45f8b4ba-5b82-4854-bf6d-4f490dfb28e6/volumes" Jul 12 19:32:12.442142 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:12.442093 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/maas-keycloak-0" event={"ID":"4e890b52-05a8-494b-946e-8ad3db3c048a","Type":"ContainerStarted","Data":"adca4a0bc85778c1d75b60695e16f013285d050c9530829ae598da3dc837bc6f"} Jul 12 19:32:12.459640 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:12.459591 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="keycloak-system/maas-keycloak-0" podStartSLOduration=2.190474906 podStartE2EDuration="2.459575852s" totalImagesPullingTime="269.100946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:32:12.457225622 +0000 UTC m=+621.139704083" watchObservedRunningTime="2026-07-12 19:32:12.459575852 +0000 UTC m=+621.142054312" Jul 12 19:32:12.786502 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:12.786390 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:12.788512 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:12.788431 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:13.785738 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:13.785688 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:14.786072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:14.785950 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:15.786101 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:15.786046 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:16.786205 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:16.786141 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:17.788686 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:17.788634 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:18.786110 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:18.786055 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:19.786035 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:19.785989 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:20.785983 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:20.785940 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:20.786346 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:20.786308 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:21.786475 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:21.786436 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:22.785781 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:22.785731 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:23.785958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:23.785906 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:24.786722 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:24.786671 2565 prober.go:120] "Probe failed" probeType="Startup" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="Get \"http://10.132.0.35:9000/health/started\": dial tcp 10.132.0.35:9000: connect: connection refused" Jul 12 19:32:25.883402 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:25.883365 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="keycloak-system/maas-keycloak-0" Jul 12 19:32:25.900764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:25.900716 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="keycloak-system/maas-keycloak-0" podUID="4e890b52-05a8-494b-946e-8ad3db3c048a" containerName="keycloak" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 12 19:32:35.890028 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:32:35.889993 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="keycloak-system/maas-keycloak-0" Jul 12 19:33:14.976327 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:14.976291 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp"] Jul 12 19:33:14.980981 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:14.980961 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:14.983162 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:14.983139 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 12 19:33:14.983162 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:14.983155 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 12 19:33:14.984168 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:14.984151 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-48spd\"" Jul 12 19:33:14.984300 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:14.984193 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 12 19:33:14.990147 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:14.990126 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp"] Jul 12 19:33:15.123764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.123728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.123935 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.123770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.123935 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.123792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb6fz\" (UniqueName: \"kubernetes.io/projected/62845992-558f-4686-96f2-5c26da67a74a-kube-api-access-bb6fz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.123935 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.123833 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.123935 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.123880 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.124069 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.123947 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62845992-558f-4686-96f2-5c26da67a74a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225108 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225065 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62845992-558f-4686-96f2-5c26da67a74a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225306 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225122 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225306 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225145 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225306 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225161 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb6fz\" (UniqueName: \"kubernetes.io/projected/62845992-558f-4686-96f2-5c26da67a74a-kube-api-access-bb6fz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225306 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225204 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225306 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225228 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225625 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225699 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225646 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.225753 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.225707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.227440 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.227398 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62845992-558f-4686-96f2-5c26da67a74a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.227742 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.227725 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62845992-558f-4686-96f2-5c26da67a74a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.232301 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.232278 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb6fz\" (UniqueName: \"kubernetes.io/projected/62845992-558f-4686-96f2-5c26da67a74a-kube-api-access-bb6fz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp\" (UID: \"62845992-558f-4686-96f2-5c26da67a74a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.291997 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.291949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:15.399630 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.399597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:33:15.427494 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.427374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp"] Jul 12 19:33:15.429778 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:33:15.429752 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62845992_558f_4686_96f2_5c26da67a74a.slice/crio-f7005104e1a2828cd19506bf527d0de2bf2d8e3e8cbafb511f39ce50f640265c WatchSource:0}: Error finding container f7005104e1a2828cd19506bf527d0de2bf2d8e3e8cbafb511f39ce50f640265c: Status 404 returned error can't find the container with id f7005104e1a2828cd19506bf527d0de2bf2d8e3e8cbafb511f39ce50f640265c Jul 12 19:33:15.430987 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.430968 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:33:15.760943 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:15.760903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" event={"ID":"62845992-558f-4686-96f2-5c26da67a74a","Type":"ContainerStarted","Data":"f7005104e1a2828cd19506bf527d0de2bf2d8e3e8cbafb511f39ce50f640265c"} Jul 12 19:33:16.767436 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:16.767391 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" event={"ID":"62845992-558f-4686-96f2-5c26da67a74a","Type":"ContainerStarted","Data":"18fd978cd4ae27164702424e076d63b897efa1b8a935938960f17f0ceb6b087c"} Jul 12 19:33:16.783065 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:16.783015 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" podStartSLOduration=1.609634234 podStartE2EDuration="2.783s" totalImagesPullingTime="1.173365766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:33:16.782499366 +0000 UTC m=+685.464977827" watchObservedRunningTime="2026-07-12 19:33:16.783 +0000 UTC m=+685.465478460" Jul 12 19:33:21.632494 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.632459 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q"] Jul 12 19:33:21.637627 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.637593 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.640339 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.640308 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 12 19:33:21.644958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.644935 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q"] Jul 12 19:33:21.784435 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.784397 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.784435 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.784439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.784669 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.784455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a91b11-9003-4d1d-9833-73026fc788b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.784669 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.784480 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.784669 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.784591 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrdh\" (UniqueName: \"kubernetes.io/projected/d7a91b11-9003-4d1d-9833-73026fc788b5-kube-api-access-svrdh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.784669 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.784644 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.885805 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.885723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.885805 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.885759 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.885805 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.885774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a91b11-9003-4d1d-9833-73026fc788b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.885805 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.885792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.886131 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.885832 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrdh\" (UniqueName: \"kubernetes.io/projected/d7a91b11-9003-4d1d-9833-73026fc788b5-kube-api-access-svrdh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.886131 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.885870 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.886279 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.886172 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.886322 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.886282 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.886322 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.886298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.888072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.888049 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7a91b11-9003-4d1d-9833-73026fc788b5-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.888283 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.888257 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7a91b11-9003-4d1d-9833-73026fc788b5-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.892483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.892464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrdh\" (UniqueName: \"kubernetes.io/projected/d7a91b11-9003-4d1d-9833-73026fc788b5-kube-api-access-svrdh\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5rc9q\" (UID: \"d7a91b11-9003-4d1d-9833-73026fc788b5\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:21.951630 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:21.951593 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:22.088750 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:22.088721 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q"] Jul 12 19:33:22.091194 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:33:22.091157 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a91b11_9003_4d1d_9833_73026fc788b5.slice/crio-0f88a0c908c783dad93d36f3550af880ee6554dc1c09d68b8003a158064d2aa0 WatchSource:0}: Error finding container 0f88a0c908c783dad93d36f3550af880ee6554dc1c09d68b8003a158064d2aa0: Status 404 returned error can't find the container with id 0f88a0c908c783dad93d36f3550af880ee6554dc1c09d68b8003a158064d2aa0 Jul 12 19:33:22.168768 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:22.168685 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:33:22.799881 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:22.799844 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" event={"ID":"d7a91b11-9003-4d1d-9833-73026fc788b5","Type":"ContainerStarted","Data":"7f3e99019ad9229235267228b90fda04ee19438ff7559e9534c8105cc6ce5453"} Jul 12 19:33:22.800380 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:22.799907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" event={"ID":"d7a91b11-9003-4d1d-9833-73026fc788b5","Type":"ContainerStarted","Data":"0f88a0c908c783dad93d36f3550af880ee6554dc1c09d68b8003a158064d2aa0"} Jul 12 19:33:22.816231 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:22.816166 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" podStartSLOduration=1.6473111280000001 podStartE2EDuration="1.816152106s" totalImagesPullingTime="168.840978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:33:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:33:22.814343757 +0000 UTC m=+691.496822215" watchObservedRunningTime="2026-07-12 19:33:22.816152106 +0000 UTC m=+691.498630566" Jul 12 19:33:25.292882 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:25.292843 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:25.293298 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:25.292912 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:25.305461 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:25.305433 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:25.824604 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:25.824576 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp" Jul 12 19:33:31.952504 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:31.952463 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:31.952885 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:31.952525 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:31.965477 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:31.965441 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:32.857481 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:32.857450 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5rc9q" Jul 12 19:33:33.672771 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:33:33.672732 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:34:08.573781 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:34:08.573700 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:34:19.774515 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:34:19.774480 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:34:30.073254 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:34:30.073220 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:35:53.579689 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:35:53.579653 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:36:02.973243 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:02.973202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:36:14.775472 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:14.775437 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:36:22.772887 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:22.772852 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:36:33.872712 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:33.872682 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:36:36.007962 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.007927 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-557cf6647c-vlzkq"] Jul 12 19:36:36.008374 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.008118 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-557cf6647c-vlzkq" podUID="886ddc55-627b-4578-af55-aa55c9ad2eb8" containerName="manager" containerID="cri-o://32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc" gracePeriod=30 Jul 12 19:36:36.266608 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.266551 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:36:36.389445 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.389416 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbzbr\" (UniqueName: \"kubernetes.io/projected/886ddc55-627b-4578-af55-aa55c9ad2eb8-kube-api-access-zbzbr\") pod \"886ddc55-627b-4578-af55-aa55c9ad2eb8\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " Jul 12 19:36:36.389635 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.389561 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/886ddc55-627b-4578-af55-aa55c9ad2eb8-webhook-cert\") pod \"886ddc55-627b-4578-af55-aa55c9ad2eb8\" (UID: \"886ddc55-627b-4578-af55-aa55c9ad2eb8\") " Jul 12 19:36:36.391587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.391552 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886ddc55-627b-4578-af55-aa55c9ad2eb8-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "886ddc55-627b-4578-af55-aa55c9ad2eb8" (UID: "886ddc55-627b-4578-af55-aa55c9ad2eb8"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:36:36.391693 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.391565 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886ddc55-627b-4578-af55-aa55c9ad2eb8-kube-api-access-zbzbr" (OuterVolumeSpecName: "kube-api-access-zbzbr") pod "886ddc55-627b-4578-af55-aa55c9ad2eb8" (UID: "886ddc55-627b-4578-af55-aa55c9ad2eb8"). InnerVolumeSpecName "kube-api-access-zbzbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:36:36.490889 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.490854 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/886ddc55-627b-4578-af55-aa55c9ad2eb8-webhook-cert\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:36:36.490889 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.490884 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzbr\" (UniqueName: \"kubernetes.io/projected/886ddc55-627b-4578-af55-aa55c9ad2eb8-kube-api-access-zbzbr\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:36:36.696549 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.696509 2565 generic.go:356] "Generic (PLEG): container finished" podID="886ddc55-627b-4578-af55-aa55c9ad2eb8" containerID="32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc" exitCode=0 Jul 12 19:36:36.696714 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.696557 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557cf6647c-vlzkq" Jul 12 19:36:36.696714 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.696597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557cf6647c-vlzkq" event={"ID":"886ddc55-627b-4578-af55-aa55c9ad2eb8","Type":"ContainerDied","Data":"32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc"} Jul 12 19:36:36.696714 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.696656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557cf6647c-vlzkq" event={"ID":"886ddc55-627b-4578-af55-aa55c9ad2eb8","Type":"ContainerDied","Data":"1bb3d437899afc9cf8dcb1d71ec8a83a419660d46cf48c1ce820a9eff9fd6c7c"} Jul 12 19:36:36.696714 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.696680 2565 scope.go:117] "RemoveContainer" containerID="32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc" Jul 12 19:36:36.705931 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.705905 2565 scope.go:117] "RemoveContainer" containerID="32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc" Jul 12 19:36:36.706157 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:36:36.706138 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc\": container with ID starting with 32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc not found: ID does not exist" containerID="32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc" Jul 12 19:36:36.706239 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.706166 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc"} err="failed to get container status \"32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc\": rpc error: code = NotFound desc = could not find container \"32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc\": container with ID starting with 32484e4054c5aa24a1897d420c5e4c7e4af08d40e0372d4e16fba06ab0f458dc not found: ID does not exist" Jul 12 19:36:36.716227 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.716203 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-557cf6647c-vlzkq"] Jul 12 19:36:36.719719 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:36.719700 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-557cf6647c-vlzkq"] Jul 12 19:36:37.474114 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.474075 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-557cf6647c-rz92f"] Jul 12 19:36:37.474487 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.474452 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="886ddc55-627b-4578-af55-aa55c9ad2eb8" containerName="manager" Jul 12 19:36:37.474487 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.474464 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="886ddc55-627b-4578-af55-aa55c9ad2eb8" containerName="manager" Jul 12 19:36:37.474561 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.474535 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="886ddc55-627b-4578-af55-aa55c9ad2eb8" containerName="manager" Jul 12 19:36:37.478924 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.478903 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.482047 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.482021 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sr8w4\"" Jul 12 19:36:37.482047 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.482021 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 12 19:36:37.482217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.482035 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 12 19:36:37.493798 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.493766 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557cf6647c-rz92f"] Jul 12 19:36:37.600352 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.600315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7142d987-c46d-4bdc-92b9-72cd1c35f76a-webhook-cert\") pod \"maas-controller-557cf6647c-rz92f\" (UID: \"7142d987-c46d-4bdc-92b9-72cd1c35f76a\") " pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.600505 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.600370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wflns\" (UniqueName: \"kubernetes.io/projected/7142d987-c46d-4bdc-92b9-72cd1c35f76a-kube-api-access-wflns\") pod \"maas-controller-557cf6647c-rz92f\" (UID: \"7142d987-c46d-4bdc-92b9-72cd1c35f76a\") " pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.701669 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.701634 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7142d987-c46d-4bdc-92b9-72cd1c35f76a-webhook-cert\") pod \"maas-controller-557cf6647c-rz92f\" (UID: \"7142d987-c46d-4bdc-92b9-72cd1c35f76a\") " pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.701861 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.701686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wflns\" (UniqueName: \"kubernetes.io/projected/7142d987-c46d-4bdc-92b9-72cd1c35f76a-kube-api-access-wflns\") pod \"maas-controller-557cf6647c-rz92f\" (UID: \"7142d987-c46d-4bdc-92b9-72cd1c35f76a\") " pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.704140 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.704113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7142d987-c46d-4bdc-92b9-72cd1c35f76a-webhook-cert\") pod \"maas-controller-557cf6647c-rz92f\" (UID: \"7142d987-c46d-4bdc-92b9-72cd1c35f76a\") " pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.709590 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.709565 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wflns\" (UniqueName: \"kubernetes.io/projected/7142d987-c46d-4bdc-92b9-72cd1c35f76a-kube-api-access-wflns\") pod \"maas-controller-557cf6647c-rz92f\" (UID: \"7142d987-c46d-4bdc-92b9-72cd1c35f76a\") " pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.817730 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.817642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:37.863733 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.863700 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="886ddc55-627b-4578-af55-aa55c9ad2eb8" path="/var/lib/kubelet/pods/886ddc55-627b-4578-af55-aa55c9ad2eb8/volumes" Jul 12 19:36:37.947154 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:37.947129 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-557cf6647c-rz92f"] Jul 12 19:36:37.949114 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:36:37.949080 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7142d987_c46d_4bdc_92b9_72cd1c35f76a.slice/crio-f71a9fcb52a05e5e5837bbf34316a42f209adca48f89282023c3d2d29c525a7a WatchSource:0}: Error finding container f71a9fcb52a05e5e5837bbf34316a42f209adca48f89282023c3d2d29c525a7a: Status 404 returned error can't find the container with id f71a9fcb52a05e5e5837bbf34316a42f209adca48f89282023c3d2d29c525a7a Jul 12 19:36:38.708280 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:38.708239 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557cf6647c-rz92f" event={"ID":"7142d987-c46d-4bdc-92b9-72cd1c35f76a","Type":"ContainerStarted","Data":"e78543f53afa40af62f9c3932d4864d81d243171eb9a593f129db6d3d2f95600"} Jul 12 19:36:38.708751 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:38.708304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-557cf6647c-rz92f" event={"ID":"7142d987-c46d-4bdc-92b9-72cd1c35f76a","Type":"ContainerStarted","Data":"f71a9fcb52a05e5e5837bbf34316a42f209adca48f89282023c3d2d29c525a7a"} Jul 12 19:36:38.708751 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:38.708361 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:38.722548 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:38.722503 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-557cf6647c-rz92f" podStartSLOduration=1.164866679 podStartE2EDuration="1.722488828s" totalImagesPullingTime="557.622149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:36:38.722296858 +0000 UTC m=+887.404775318" watchObservedRunningTime="2026-07-12 19:36:38.722488828 +0000 UTC m=+887.404967287" Jul 12 19:36:49.717849 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:49.717777 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-557cf6647c-rz92f" Jul 12 19:36:51.792576 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:51.792542 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:36:51.793602 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:51.793574 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:36:51.796405 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:51.796387 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:36:51.797488 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:36:51.797471 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:37:02.371622 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:37:02.371586 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:37:11.772776 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:37:11.772741 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:37:44.602405 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:37:44.602362 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:37:53.672197 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:37:53.672141 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:38:03.868781 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:38:03.868745 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:38:09.577995 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:38:09.577956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:38:20.076753 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:38:20.076660 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:38:32.620111 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:38:32.620058 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:38:43.171657 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:38:43.171613 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:38:52.776526 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:38:52.776485 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:39:15.071291 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:39:15.071252 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:39:27.183355 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:39:27.183319 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:39:50.278099 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:39:50.278020 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:40:00.977255 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:40:00.977209 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:40:15.470853 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:40:15.470815 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:40:27.174364 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:40:27.174327 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:40:36.875727 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:40:36.875694 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:40:47.771837 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:40:47.771801 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:41:51.821159 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:41:51.821129 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:41:51.821653 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:41:51.821129 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:41:51.825041 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:41:51.825019 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:41:51.825171 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:41:51.825060 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:42:42.471471 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:42:42.471431 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:42:54.179281 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:42:54.179194 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:43:02.585589 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:43:02.585553 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:43:15.375674 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:43:15.375638 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:43:27.174954 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:43:27.174922 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:43:36.970797 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:43:36.970762 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:43:46.081573 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:43:46.081537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:43:55.572776 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:43:55.572738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:44:07.672167 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:44:07.672131 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:44:16.677912 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:44:16.677825 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:44:27.174904 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:44:27.174872 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:44:51.578656 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:44:51.578623 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:44:59.071208 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:44:59.071161 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:45:00.138110 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.138079 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29731425-bmhlc"] Jul 12 19:45:00.141753 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.141729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" Jul 12 19:45:00.157814 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.157785 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29731425-bmhlc"] Jul 12 19:45:00.248483 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.248448 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hvs\" (UniqueName: \"kubernetes.io/projected/a7386590-582f-48b4-bdd7-769be41b6815-kube-api-access-b9hvs\") pod \"maas-api-key-cleanup-29731425-bmhlc\" (UID: \"a7386590-582f-48b4-bdd7-769be41b6815\") " pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" Jul 12 19:45:00.349152 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.349118 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hvs\" (UniqueName: \"kubernetes.io/projected/a7386590-582f-48b4-bdd7-769be41b6815-kube-api-access-b9hvs\") pod \"maas-api-key-cleanup-29731425-bmhlc\" (UID: \"a7386590-582f-48b4-bdd7-769be41b6815\") " pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" Jul 12 19:45:00.357324 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.357293 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hvs\" (UniqueName: \"kubernetes.io/projected/a7386590-582f-48b4-bdd7-769be41b6815-kube-api-access-b9hvs\") pod \"maas-api-key-cleanup-29731425-bmhlc\" (UID: \"a7386590-582f-48b4-bdd7-769be41b6815\") " pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" Jul 12 19:45:00.451649 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.451566 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" Jul 12 19:45:00.605472 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.605447 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29731425-bmhlc"] Jul 12 19:45:00.607884 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:45:00.607859 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7386590_582f_48b4_bdd7_769be41b6815.slice/crio-fcedbe090ead0591db7319655b84c4866a7ae85c6fd33c7f9e6db1f18ed03b49 WatchSource:0}: Error finding container fcedbe090ead0591db7319655b84c4866a7ae85c6fd33c7f9e6db1f18ed03b49: Status 404 returned error can't find the container with id fcedbe090ead0591db7319655b84c4866a7ae85c6fd33c7f9e6db1f18ed03b49 Jul 12 19:45:00.610158 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:00.610144 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:45:01.045053 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:01.045018 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" event={"ID":"a7386590-582f-48b4-bdd7-769be41b6815","Type":"ContainerStarted","Data":"fcedbe090ead0591db7319655b84c4866a7ae85c6fd33c7f9e6db1f18ed03b49"} Jul 12 19:45:04.061956 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:04.061916 2565 generic.go:356] "Generic (PLEG): container finished" podID="a7386590-582f-48b4-bdd7-769be41b6815" containerID="1519c825de40b26959f4f990d7a6050cecfda2bd5e839ba43becbbdc6ffd1087" exitCode=0 Jul 12 19:45:04.062359 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:04.061955 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" event={"ID":"a7386590-582f-48b4-bdd7-769be41b6815","Type":"ContainerDied","Data":"1519c825de40b26959f4f990d7a6050cecfda2bd5e839ba43becbbdc6ffd1087"} Jul 12 19:45:05.203540 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:05.203514 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" Jul 12 19:45:05.294944 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:05.294903 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9hvs\" (UniqueName: \"kubernetes.io/projected/a7386590-582f-48b4-bdd7-769be41b6815-kube-api-access-b9hvs\") pod \"a7386590-582f-48b4-bdd7-769be41b6815\" (UID: \"a7386590-582f-48b4-bdd7-769be41b6815\") " Jul 12 19:45:05.296968 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:05.296941 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7386590-582f-48b4-bdd7-769be41b6815-kube-api-access-b9hvs" (OuterVolumeSpecName: "kube-api-access-b9hvs") pod "a7386590-582f-48b4-bdd7-769be41b6815" (UID: "a7386590-582f-48b4-bdd7-769be41b6815"). InnerVolumeSpecName "kube-api-access-b9hvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:45:05.395933 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:05.395891 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hvs\" (UniqueName: \"kubernetes.io/projected/a7386590-582f-48b4-bdd7-769be41b6815-kube-api-access-b9hvs\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:45:06.074217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:06.074169 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" Jul 12 19:45:06.074217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:06.074169 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29731425-bmhlc" event={"ID":"a7386590-582f-48b4-bdd7-769be41b6815","Type":"ContainerDied","Data":"fcedbe090ead0591db7319655b84c4866a7ae85c6fd33c7f9e6db1f18ed03b49"} Jul 12 19:45:06.074424 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:06.074249 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcedbe090ead0591db7319655b84c4866a7ae85c6fd33c7f9e6db1f18ed03b49" Jul 12 19:45:08.675991 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:08.675955 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:45:21.273310 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:21.273279 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:45:37.672428 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:37.672398 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:45:48.528480 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:48.528400 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:45:58.074137 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:45:58.074101 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:46:08.179650 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:08.179609 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:46:17.376400 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:17.376363 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:46:27.175963 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:27.175925 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:46:37.777338 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:37.777305 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:46:48.984866 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:48.984829 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:46:51.856984 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:51.856955 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:46:51.857481 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:51.857021 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:46:51.862088 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:51.862064 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:46:51.864679 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:51.864658 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:46:59.381091 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:46:59.381053 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:47:10.865641 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:47:10.865601 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:47:23.376895 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:47:23.376809 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:47:31.468247 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:47:31.468204 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:47:43.067445 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:47:43.067410 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:47:54.874204 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:47:54.874145 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:48:07.271782 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:07.271747 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:48:16.670983 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:16.670944 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:48:36.673236 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:36.673198 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:48:43.516006 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.515970 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx"] Jul 12 19:48:43.517090 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.517064 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7386590-582f-48b4-bdd7-769be41b6815" containerName="cleanup" Jul 12 19:48:43.517090 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.517089 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7386590-582f-48b4-bdd7-769be41b6815" containerName="cleanup" Jul 12 19:48:43.517261 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.517203 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7386590-582f-48b4-bdd7-769be41b6815" containerName="cleanup" Jul 12 19:48:43.520040 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.520022 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.522371 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.522353 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-a4a89c5e-serving-cert\"" Jul 12 19:48:43.531720 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.531697 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx"] Jul 12 19:48:43.680488 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.680440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.680488 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.680493 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd6v5\" (UniqueName: \"kubernetes.io/projected/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-kube-api-access-sd6v5\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.680688 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.680566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.782130 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.782046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.782130 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.782098 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd6v5\" (UniqueName: \"kubernetes.io/projected/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-kube-api-access-sd6v5\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.782327 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.782143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.782327 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:48:43.782204 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-a4a89c5e-serving-cert: secret "maas-api-e2e-ait-a4a89c5e-serving-cert" not found Jul 12 19:48:43.782327 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:48:43.782280 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls podName:4b8a5b69-9c1b-4be6-949b-8a34943ab9e4 nodeName:}" failed. No retries permitted until 2026-07-12 19:48:44.282261703 +0000 UTC m=+1612.964740148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls") pod "maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" (UID: "4b8a5b69-9c1b-4be6-949b-8a34943ab9e4") : secret "maas-api-e2e-ait-a4a89c5e-serving-cert" not found Jul 12 19:48:43.782905 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.782885 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-openshift-service-ca\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:43.793966 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:43.793921 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd6v5\" (UniqueName: \"kubernetes.io/projected/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-kube-api-access-sd6v5\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:44.285729 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:44.285639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:44.288091 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:44.288066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls\") pod \"maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:44.432858 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:44.432823 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:44.572691 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:44.572667 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx"] Jul 12 19:48:44.574763 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:48:44.574734 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8a5b69_9c1b_4be6_949b_8a34943ab9e4.slice/crio-f4f57904cebf51849aa73f86ecf13ce31b06b66d64559c3b3a7d39a0268b2d12 WatchSource:0}: Error finding container f4f57904cebf51849aa73f86ecf13ce31b06b66d64559c3b3a7d39a0268b2d12: Status 404 returned error can't find the container with id f4f57904cebf51849aa73f86ecf13ce31b06b66d64559c3b3a7d39a0268b2d12 Jul 12 19:48:45.101367 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:45.101338 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" event={"ID":"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4","Type":"ContainerStarted","Data":"f4f57904cebf51849aa73f86ecf13ce31b06b66d64559c3b3a7d39a0268b2d12"} Jul 12 19:48:46.107365 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:46.107331 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" event={"ID":"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4","Type":"ContainerStarted","Data":"b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb"} Jul 12 19:48:46.107727 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:46.107457 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:46.123555 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:46.123507 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" podStartSLOduration=2.600291301 podStartE2EDuration="3.123491814s" totalImagesPullingTime="523.200513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:48:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:48:46.122945393 +0000 UTC m=+1614.805423854" watchObservedRunningTime="2026-07-12 19:48:46.123491814 +0000 UTC m=+1614.805970274" Jul 12 19:48:47.078167 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:47.078130 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:48:48.726808 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:48.726776 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx"] Jul 12 19:48:48.727376 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:48.726994 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" podUID="4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" containerName="maas-api" containerID="cri-o://b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb" gracePeriod=30 Jul 12 19:48:48.979228 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:48.979147 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:49.122025 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.121993 2565 generic.go:356] "Generic (PLEG): container finished" podID="4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" containerID="b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb" exitCode=0 Jul 12 19:48:49.122206 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.122035 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" Jul 12 19:48:49.122206 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.122075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" event={"ID":"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4","Type":"ContainerDied","Data":"b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb"} Jul 12 19:48:49.122206 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.122142 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx" event={"ID":"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4","Type":"ContainerDied","Data":"f4f57904cebf51849aa73f86ecf13ce31b06b66d64559c3b3a7d39a0268b2d12"} Jul 12 19:48:49.122206 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.122171 2565 scope.go:117] "RemoveContainer" containerID="b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb" Jul 12 19:48:49.129410 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.129388 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls\") pod \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " Jul 12 19:48:49.129523 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.129419 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-openshift-service-ca\") pod \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " Jul 12 19:48:49.129583 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.129522 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd6v5\" (UniqueName: \"kubernetes.io/projected/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-kube-api-access-sd6v5\") pod \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\" (UID: \"4b8a5b69-9c1b-4be6-949b-8a34943ab9e4\") " Jul 12 19:48:49.129834 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.129802 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" (UID: "4b8a5b69-9c1b-4be6-949b-8a34943ab9e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:48:49.131009 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.130984 2565 scope.go:117] "RemoveContainer" containerID="b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb" Jul 12 19:48:49.131338 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:48:49.131309 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb\": container with ID starting with b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb not found: ID does not exist" containerID="b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb" Jul 12 19:48:49.131416 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.131349 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb"} err="failed to get container status \"b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb\": rpc error: code = NotFound desc = could not find container \"b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb\": container with ID starting with b06664cd8e6c2c6855d54c0e1044d5fb844705abdbc6cfdc9d5acef9d86d55cb not found: ID does not exist" Jul 12 19:48:49.131542 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.131523 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" (UID: "4b8a5b69-9c1b-4be6-949b-8a34943ab9e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:48:49.131631 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.131614 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-kube-api-access-sd6v5" (OuterVolumeSpecName: "kube-api-access-sd6v5") pod "4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" (UID: "4b8a5b69-9c1b-4be6-949b-8a34943ab9e4"). InnerVolumeSpecName "kube-api-access-sd6v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:48:49.231278 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.231159 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:48:49.231278 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.231224 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:48:49.231278 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.231234 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd6v5\" (UniqueName: \"kubernetes.io/projected/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4-kube-api-access-sd6v5\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:48:49.449031 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.448996 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx"] Jul 12 19:48:49.452253 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.452230 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-a4a89c5e-6b8c9b7d89-77dhx"] Jul 12 19:48:49.861959 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:49.861925 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" path="/var/lib/kubelet/pods/4b8a5b69-9c1b-4be6-949b-8a34943ab9e4/volumes" Jul 12 19:48:55.733634 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.733600 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md"] Jul 12 19:48:55.734137 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.734118 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" containerName="maas-api" Jul 12 19:48:55.734233 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.734140 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" containerName="maas-api" Jul 12 19:48:55.734302 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.734261 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8a5b69-9c1b-4be6-949b-8a34943ab9e4" containerName="maas-api" Jul 12 19:48:55.737330 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.737309 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.739532 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.739513 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-ce5aeab4-serving-cert\"" Jul 12 19:48:55.753115 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.753089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md"] Jul 12 19:48:55.785577 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.785532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b87f25-3060-4e87-8e07-03fca368a8f7-openshift-service-ca\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.785761 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.785601 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.785761 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.785626 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mkf\" (UniqueName: \"kubernetes.io/projected/b1b87f25-3060-4e87-8e07-03fca368a8f7-kube-api-access-p2mkf\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.886424 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.886388 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mkf\" (UniqueName: \"kubernetes.io/projected/b1b87f25-3060-4e87-8e07-03fca368a8f7-kube-api-access-p2mkf\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.886602 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.886440 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b87f25-3060-4e87-8e07-03fca368a8f7-openshift-service-ca\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.886602 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.886503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.886602 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:48:55.886597 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-ce5aeab4-serving-cert: secret "maas-api-e2e-ait-ce5aeab4-serving-cert" not found Jul 12 19:48:55.886761 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:48:55.886669 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls podName:b1b87f25-3060-4e87-8e07-03fca368a8f7 nodeName:}" failed. No retries permitted until 2026-07-12 19:48:56.386650397 +0000 UTC m=+1625.069128838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls") pod "maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" (UID: "b1b87f25-3060-4e87-8e07-03fca368a8f7") : secret "maas-api-e2e-ait-ce5aeab4-serving-cert" not found Jul 12 19:48:55.887191 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.887158 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b87f25-3060-4e87-8e07-03fca368a8f7-openshift-service-ca\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:55.894371 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:55.894349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mkf\" (UniqueName: \"kubernetes.io/projected/b1b87f25-3060-4e87-8e07-03fca368a8f7-kube-api-access-p2mkf\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:56.390924 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:56.390892 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:56.393443 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:56.393421 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls\") pod \"maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:56.648706 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:56.648620 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:56.797788 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:56.797764 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md"] Jul 12 19:48:56.799985 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:48:56.799957 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b87f25_3060_4e87_8e07_03fca368a8f7.slice/crio-d3e87e08e4ac0d19476ea81f63130cfeacff38cedf226d483c902e574852c338 WatchSource:0}: Error finding container d3e87e08e4ac0d19476ea81f63130cfeacff38cedf226d483c902e574852c338: Status 404 returned error can't find the container with id d3e87e08e4ac0d19476ea81f63130cfeacff38cedf226d483c902e574852c338 Jul 12 19:48:57.163435 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:57.163405 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" event={"ID":"b1b87f25-3060-4e87-8e07-03fca368a8f7","Type":"ContainerStarted","Data":"d3e87e08e4ac0d19476ea81f63130cfeacff38cedf226d483c902e574852c338"} Jul 12 19:48:57.675724 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:57.675689 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:48:58.169226 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:58.169193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" event={"ID":"b1b87f25-3060-4e87-8e07-03fca368a8f7","Type":"ContainerStarted","Data":"c37f951af65ba8384d0abb6ea1814c6ea101b35edc87da6898a36c452b172834"} Jul 12 19:48:58.169580 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:58.169313 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:48:58.184819 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:48:58.184773 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" podStartSLOduration=2.841902307 podStartE2EDuration="3.184757513s" totalImagesPullingTime="342.855206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:48:58.18373181 +0000 UTC m=+1626.866210271" watchObservedRunningTime="2026-07-12 19:48:58.184757513 +0000 UTC m=+1626.867235973" Jul 12 19:49:01.005036 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.004995 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md"] Jul 12 19:49:01.005520 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.005273 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" podUID="b1b87f25-3060-4e87-8e07-03fca368a8f7" containerName="maas-api" containerID="cri-o://c37f951af65ba8384d0abb6ea1814c6ea101b35edc87da6898a36c452b172834" gracePeriod=30 Jul 12 19:49:01.186147 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.186112 2565 generic.go:356] "Generic (PLEG): container finished" podID="b1b87f25-3060-4e87-8e07-03fca368a8f7" containerID="c37f951af65ba8384d0abb6ea1814c6ea101b35edc87da6898a36c452b172834" exitCode=0 Jul 12 19:49:01.186315 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.186296 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" event={"ID":"b1b87f25-3060-4e87-8e07-03fca368a8f7","Type":"ContainerDied","Data":"c37f951af65ba8384d0abb6ea1814c6ea101b35edc87da6898a36c452b172834"} Jul 12 19:49:01.255347 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.255289 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:49:01.336321 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.336284 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2mkf\" (UniqueName: \"kubernetes.io/projected/b1b87f25-3060-4e87-8e07-03fca368a8f7-kube-api-access-p2mkf\") pod \"b1b87f25-3060-4e87-8e07-03fca368a8f7\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " Jul 12 19:49:01.336511 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.336337 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls\") pod \"b1b87f25-3060-4e87-8e07-03fca368a8f7\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " Jul 12 19:49:01.336511 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.336455 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b87f25-3060-4e87-8e07-03fca368a8f7-openshift-service-ca\") pod \"b1b87f25-3060-4e87-8e07-03fca368a8f7\" (UID: \"b1b87f25-3060-4e87-8e07-03fca368a8f7\") " Jul 12 19:49:01.336772 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.336744 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b87f25-3060-4e87-8e07-03fca368a8f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1b87f25-3060-4e87-8e07-03fca368a8f7" (UID: "b1b87f25-3060-4e87-8e07-03fca368a8f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:49:01.338519 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.338496 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1b87f25-3060-4e87-8e07-03fca368a8f7" (UID: "b1b87f25-3060-4e87-8e07-03fca368a8f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:49:01.338586 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.338500 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b87f25-3060-4e87-8e07-03fca368a8f7-kube-api-access-p2mkf" (OuterVolumeSpecName: "kube-api-access-p2mkf") pod "b1b87f25-3060-4e87-8e07-03fca368a8f7" (UID: "b1b87f25-3060-4e87-8e07-03fca368a8f7"). InnerVolumeSpecName "kube-api-access-p2mkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:01.438052 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.438017 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b87f25-3060-4e87-8e07-03fca368a8f7-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:49:01.438052 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.438047 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2mkf\" (UniqueName: \"kubernetes.io/projected/b1b87f25-3060-4e87-8e07-03fca368a8f7-kube-api-access-p2mkf\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:49:01.438052 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:01.438056 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b87f25-3060-4e87-8e07-03fca368a8f7-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:49:02.191830 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:02.191736 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" event={"ID":"b1b87f25-3060-4e87-8e07-03fca368a8f7","Type":"ContainerDied","Data":"d3e87e08e4ac0d19476ea81f63130cfeacff38cedf226d483c902e574852c338"} Jul 12 19:49:02.191830 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:02.191765 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md" Jul 12 19:49:02.191830 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:02.191802 2565 scope.go:117] "RemoveContainer" containerID="c37f951af65ba8384d0abb6ea1814c6ea101b35edc87da6898a36c452b172834" Jul 12 19:49:02.207131 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:02.207107 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md"] Jul 12 19:49:02.209402 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:02.209381 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-ce5aeab4-8c48cbb9-d86md"] Jul 12 19:49:03.862425 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:03.862394 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b87f25-3060-4e87-8e07-03fca368a8f7" path="/var/lib/kubelet/pods/b1b87f25-3060-4e87-8e07-03fca368a8f7/volumes" Jul 12 19:49:14.452135 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.452098 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5"] Jul 12 19:49:14.452663 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.452643 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1b87f25-3060-4e87-8e07-03fca368a8f7" containerName="maas-api" Jul 12 19:49:14.452663 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.452663 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1b87f25-3060-4e87-8e07-03fca368a8f7" containerName="maas-api" Jul 12 19:49:14.452800 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.452777 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1b87f25-3060-4e87-8e07-03fca368a8f7" containerName="maas-api" Jul 12 19:49:14.457119 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.457098 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.459610 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.459587 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-derive-771d3436-serving-cert\"" Jul 12 19:49:14.465463 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.465241 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5"] Jul 12 19:49:14.548529 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.548492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f63965-ee1d-4723-9eb0-c86ed820be85-maas-api-tls\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.548695 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.548541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b9f63965-ee1d-4723-9eb0-c86ed820be85-kube-api-access-r8vgs\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.548695 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.548605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f63965-ee1d-4723-9eb0-c86ed820be85-openshift-service-ca\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.649470 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.649431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f63965-ee1d-4723-9eb0-c86ed820be85-maas-api-tls\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.649648 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.649482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b9f63965-ee1d-4723-9eb0-c86ed820be85-kube-api-access-r8vgs\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.649648 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.649507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f63965-ee1d-4723-9eb0-c86ed820be85-openshift-service-ca\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.650133 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.650114 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f63965-ee1d-4723-9eb0-c86ed820be85-openshift-service-ca\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.651928 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.651905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f63965-ee1d-4723-9eb0-c86ed820be85-maas-api-tls\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.657148 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.657110 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b9f63965-ee1d-4723-9eb0-c86ed820be85-kube-api-access-r8vgs\") pod \"maas-api-e2e-derive-771d3436-57c79467fc-dwhb5\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.770428 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.770339 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:14.906342 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:14.906313 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5"] Jul 12 19:49:14.908255 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:49:14.908224 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f63965_ee1d_4723_9eb0_c86ed820be85.slice/crio-9b9f9dac203f1f88173d123d97a988245d3087657ce528c25af7ca8752593013 WatchSource:0}: Error finding container 9b9f9dac203f1f88173d123d97a988245d3087657ce528c25af7ca8752593013: Status 404 returned error can't find the container with id 9b9f9dac203f1f88173d123d97a988245d3087657ce528c25af7ca8752593013 Jul 12 19:49:15.257352 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:15.257299 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" event={"ID":"b9f63965-ee1d-4723-9eb0-c86ed820be85","Type":"ContainerStarted","Data":"9b9f9dac203f1f88173d123d97a988245d3087657ce528c25af7ca8752593013"} Jul 12 19:49:16.264100 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:16.264061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" event={"ID":"b9f63965-ee1d-4723-9eb0-c86ed820be85","Type":"ContainerStarted","Data":"dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5"} Jul 12 19:49:16.264539 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:16.264143 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:16.280756 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:16.280704 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" podStartSLOduration=1.9532229349999999 podStartE2EDuration="2.280687998s" totalImagesPullingTime="327.465063ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:49:16.278761705 +0000 UTC m=+1644.961240177" watchObservedRunningTime="2026-07-12 19:49:16.280687998 +0000 UTC m=+1644.963166457" Jul 12 19:49:18.909463 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:18.909428 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5"] Jul 12 19:49:18.909886 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:18.909666 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" podUID="b9f63965-ee1d-4723-9eb0-c86ed820be85" containerName="maas-api" containerID="cri-o://dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5" gracePeriod=30 Jul 12 19:49:19.160895 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.160829 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:19.280553 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.280515 2565 generic.go:356] "Generic (PLEG): container finished" podID="b9f63965-ee1d-4723-9eb0-c86ed820be85" containerID="dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5" exitCode=0 Jul 12 19:49:19.280553 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.280555 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" Jul 12 19:49:19.280764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.280597 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" event={"ID":"b9f63965-ee1d-4723-9eb0-c86ed820be85","Type":"ContainerDied","Data":"dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5"} Jul 12 19:49:19.280764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.280668 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5" event={"ID":"b9f63965-ee1d-4723-9eb0-c86ed820be85","Type":"ContainerDied","Data":"9b9f9dac203f1f88173d123d97a988245d3087657ce528c25af7ca8752593013"} Jul 12 19:49:19.280764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.280700 2565 scope.go:117] "RemoveContainer" containerID="dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5" Jul 12 19:49:19.289718 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.289699 2565 scope.go:117] "RemoveContainer" containerID="dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5" Jul 12 19:49:19.289971 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:49:19.289950 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5\": container with ID starting with dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5 not found: ID does not exist" containerID="dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5" Jul 12 19:49:19.290010 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.289984 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5"} err="failed to get container status \"dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5\": rpc error: code = NotFound desc = could not find container \"dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5\": container with ID starting with dbb7d8a9f5a0af75ba233d422ff7cdf3327ffcb4f0bda3649f7a0574d4261ce5 not found: ID does not exist" Jul 12 19:49:19.298413 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.298392 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f63965-ee1d-4723-9eb0-c86ed820be85-maas-api-tls\") pod \"b9f63965-ee1d-4723-9eb0-c86ed820be85\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " Jul 12 19:49:19.298497 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.298428 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b9f63965-ee1d-4723-9eb0-c86ed820be85-kube-api-access-r8vgs\") pod \"b9f63965-ee1d-4723-9eb0-c86ed820be85\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " Jul 12 19:49:19.298497 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.298490 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f63965-ee1d-4723-9eb0-c86ed820be85-openshift-service-ca\") pod \"b9f63965-ee1d-4723-9eb0-c86ed820be85\" (UID: \"b9f63965-ee1d-4723-9eb0-c86ed820be85\") " Jul 12 19:49:19.298837 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.298815 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f63965-ee1d-4723-9eb0-c86ed820be85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9f63965-ee1d-4723-9eb0-c86ed820be85" (UID: "b9f63965-ee1d-4723-9eb0-c86ed820be85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:49:19.300432 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.300412 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f63965-ee1d-4723-9eb0-c86ed820be85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9f63965-ee1d-4723-9eb0-c86ed820be85" (UID: "b9f63965-ee1d-4723-9eb0-c86ed820be85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:49:19.300510 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.300454 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f63965-ee1d-4723-9eb0-c86ed820be85-kube-api-access-r8vgs" (OuterVolumeSpecName: "kube-api-access-r8vgs") pod "b9f63965-ee1d-4723-9eb0-c86ed820be85" (UID: "b9f63965-ee1d-4723-9eb0-c86ed820be85"). InnerVolumeSpecName "kube-api-access-r8vgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:49:19.399958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.399923 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9f63965-ee1d-4723-9eb0-c86ed820be85-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:49:19.399958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.399954 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8vgs\" (UniqueName: \"kubernetes.io/projected/b9f63965-ee1d-4723-9eb0-c86ed820be85-kube-api-access-r8vgs\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:49:19.399958 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.399964 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f63965-ee1d-4723-9eb0-c86ed820be85-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:49:19.601646 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.601618 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5"] Jul 12 19:49:19.606055 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.606025 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-derive-771d3436-57c79467fc-dwhb5"] Jul 12 19:49:19.861248 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:49:19.861139 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f63965-ee1d-4723-9eb0-c86ed820be85" path="/var/lib/kubelet/pods/b9f63965-ee1d-4723-9eb0-c86ed820be85/volumes" Jul 12 19:50:52.868934 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:50:52.868901 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:51:00.471344 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:00.471308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:51:09.295609 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.295576 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98"] Jul 12 19:51:09.295990 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.295940 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f63965-ee1d-4723-9eb0-c86ed820be85" containerName="maas-api" Jul 12 19:51:09.295990 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.295951 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f63965-ee1d-4723-9eb0-c86ed820be85" containerName="maas-api" Jul 12 19:51:09.296080 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.296021 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f63965-ee1d-4723-9eb0-c86ed820be85" containerName="maas-api" Jul 12 19:51:09.299133 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.299096 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.301570 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.301552 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-41e17b7d-serving-cert\"" Jul 12 19:51:09.310706 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.310681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98"] Jul 12 19:51:09.463556 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.463520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe36e3f5-94f3-48ec-802c-c094ec003d08-openshift-service-ca\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.463556 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.463555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xzg\" (UniqueName: \"kubernetes.io/projected/fe36e3f5-94f3-48ec-802c-c094ec003d08-kube-api-access-b9xzg\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.463754 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.463605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe36e3f5-94f3-48ec-802c-c094ec003d08-maas-api-tls\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.479689 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.479652 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:51:09.564839 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.564741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe36e3f5-94f3-48ec-802c-c094ec003d08-openshift-service-ca\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.564839 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.564782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xzg\" (UniqueName: \"kubernetes.io/projected/fe36e3f5-94f3-48ec-802c-c094ec003d08-kube-api-access-b9xzg\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.565066 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.564854 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe36e3f5-94f3-48ec-802c-c094ec003d08-maas-api-tls\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.565563 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.565536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe36e3f5-94f3-48ec-802c-c094ec003d08-openshift-service-ca\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.567342 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.567324 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe36e3f5-94f3-48ec-802c-c094ec003d08-maas-api-tls\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.572423 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.572400 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xzg\" (UniqueName: \"kubernetes.io/projected/fe36e3f5-94f3-48ec-802c-c094ec003d08-kube-api-access-b9xzg\") pod \"maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.611155 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.611120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:09.752595 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.752556 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98"] Jul 12 19:51:09.754425 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:51:09.754399 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe36e3f5_94f3_48ec_802c_c094ec003d08.slice/crio-705f431b0fa1a5210dd7424d6995a36f1c50800e4b412c99bed5700bddd87bed WatchSource:0}: Error finding container 705f431b0fa1a5210dd7424d6995a36f1c50800e4b412c99bed5700bddd87bed: Status 404 returned error can't find the container with id 705f431b0fa1a5210dd7424d6995a36f1c50800e4b412c99bed5700bddd87bed Jul 12 19:51:09.755956 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.755932 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:51:09.799194 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:09.799143 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" event={"ID":"fe36e3f5-94f3-48ec-802c-c094ec003d08","Type":"ContainerStarted","Data":"705f431b0fa1a5210dd7424d6995a36f1c50800e4b412c99bed5700bddd87bed"} Jul 12 19:51:10.805528 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:10.805491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" event={"ID":"fe36e3f5-94f3-48ec-802c-c094ec003d08","Type":"ContainerStarted","Data":"3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed"} Jul 12 19:51:10.805978 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:10.805608 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:10.822221 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:10.822140 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" podStartSLOduration=1.458671485 podStartE2EDuration="1.822126059s" totalImagesPullingTime="363.454574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:51:10.820918919 +0000 UTC m=+1759.503397379" watchObservedRunningTime="2026-07-12 19:51:10.822126059 +0000 UTC m=+1759.504604542" Jul 12 19:51:16.815903 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:16.815867 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:18.975528 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:18.975493 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:51:21.183435 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.183386 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98"] Jul 12 19:51:21.183916 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.183669 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" podUID="fe36e3f5-94f3-48ec-802c-c094ec003d08" containerName="maas-api" containerID="cri-o://3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed" gracePeriod=30 Jul 12 19:51:21.438682 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.438619 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:21.570720 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.570689 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe36e3f5-94f3-48ec-802c-c094ec003d08-openshift-service-ca\") pod \"fe36e3f5-94f3-48ec-802c-c094ec003d08\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " Jul 12 19:51:21.570913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.570758 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe36e3f5-94f3-48ec-802c-c094ec003d08-maas-api-tls\") pod \"fe36e3f5-94f3-48ec-802c-c094ec003d08\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " Jul 12 19:51:21.570913 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.570780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xzg\" (UniqueName: \"kubernetes.io/projected/fe36e3f5-94f3-48ec-802c-c094ec003d08-kube-api-access-b9xzg\") pod \"fe36e3f5-94f3-48ec-802c-c094ec003d08\" (UID: \"fe36e3f5-94f3-48ec-802c-c094ec003d08\") " Jul 12 19:51:21.571126 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.571098 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe36e3f5-94f3-48ec-802c-c094ec003d08-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe36e3f5-94f3-48ec-802c-c094ec003d08" (UID: "fe36e3f5-94f3-48ec-802c-c094ec003d08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:51:21.572922 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.572887 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe36e3f5-94f3-48ec-802c-c094ec003d08-kube-api-access-b9xzg" (OuterVolumeSpecName: "kube-api-access-b9xzg") pod "fe36e3f5-94f3-48ec-802c-c094ec003d08" (UID: "fe36e3f5-94f3-48ec-802c-c094ec003d08"). InnerVolumeSpecName "kube-api-access-b9xzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:51:21.573025 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.572951 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe36e3f5-94f3-48ec-802c-c094ec003d08-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe36e3f5-94f3-48ec-802c-c094ec003d08" (UID: "fe36e3f5-94f3-48ec-802c-c094ec003d08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:51:21.672002 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.671970 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe36e3f5-94f3-48ec-802c-c094ec003d08-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:51:21.672002 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.672000 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xzg\" (UniqueName: \"kubernetes.io/projected/fe36e3f5-94f3-48ec-802c-c094ec003d08-kube-api-access-b9xzg\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:51:21.672002 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.672011 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe36e3f5-94f3-48ec-802c-c094ec003d08-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:51:21.859168 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.859127 2565 generic.go:356] "Generic (PLEG): container finished" podID="fe36e3f5-94f3-48ec-802c-c094ec003d08" containerID="3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed" exitCode=0 Jul 12 19:51:21.859338 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.859239 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" Jul 12 19:51:21.862585 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.862555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" event={"ID":"fe36e3f5-94f3-48ec-802c-c094ec003d08","Type":"ContainerDied","Data":"3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed"} Jul 12 19:51:21.862740 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.862594 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98" event={"ID":"fe36e3f5-94f3-48ec-802c-c094ec003d08","Type":"ContainerDied","Data":"705f431b0fa1a5210dd7424d6995a36f1c50800e4b412c99bed5700bddd87bed"} Jul 12 19:51:21.862740 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.862616 2565 scope.go:117] "RemoveContainer" containerID="3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed" Jul 12 19:51:21.872322 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.872300 2565 scope.go:117] "RemoveContainer" containerID="3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed" Jul 12 19:51:21.872558 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:51:21.872540 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed\": container with ID starting with 3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed not found: ID does not exist" containerID="3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed" Jul 12 19:51:21.872600 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.872567 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed"} err="failed to get container status \"3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed\": rpc error: code = NotFound desc = could not find container \"3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed\": container with ID starting with 3884e03c68cf0f066f84957b7107de84965d4c2b405fcda472b9e8d8173de4ed not found: ID does not exist" Jul 12 19:51:21.882396 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.882364 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98"] Jul 12 19:51:21.885994 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:21.885971 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-41e17b7d-6b98d54c4f-x7d98"] Jul 12 19:51:23.861870 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:23.861831 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe36e3f5-94f3-48ec-802c-c094ec003d08" path="/var/lib/kubelet/pods/fe36e3f5-94f3-48ec-802c-c094ec003d08/volumes" Jul 12 19:51:25.567413 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:25.567376 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:51:49.547086 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.546988 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq"] Jul 12 19:51:49.547549 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.547523 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe36e3f5-94f3-48ec-802c-c094ec003d08" containerName="maas-api" Jul 12 19:51:49.547549 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.547538 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe36e3f5-94f3-48ec-802c-c094ec003d08" containerName="maas-api" Jul 12 19:51:49.547665 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.547653 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe36e3f5-94f3-48ec-802c-c094ec003d08" containerName="maas-api" Jul 12 19:51:49.551355 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.551331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.553524 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.553499 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-mt-2cdc69a3-serving-cert\"" Jul 12 19:51:49.558926 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.558904 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq"] Jul 12 19:51:49.599414 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.599376 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e3bc30-92ea-4442-acea-37c5fb3e371c-openshift-service-ca\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.599588 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.599481 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.599588 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.599571 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vcmd\" (UniqueName: \"kubernetes.io/projected/42e3bc30-92ea-4442-acea-37c5fb3e371c-kube-api-access-7vcmd\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.700406 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.700367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vcmd\" (UniqueName: \"kubernetes.io/projected/42e3bc30-92ea-4442-acea-37c5fb3e371c-kube-api-access-7vcmd\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.700604 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.700422 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e3bc30-92ea-4442-acea-37c5fb3e371c-openshift-service-ca\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.700604 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.700468 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.700604 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:51:49.700552 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-mt-2cdc69a3-serving-cert: secret "maas-api-e2e-mt-2cdc69a3-serving-cert" not found Jul 12 19:51:49.700604 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:51:49.700599 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls podName:42e3bc30-92ea-4442-acea-37c5fb3e371c nodeName:}" failed. No retries permitted until 2026-07-12 19:51:50.200583984 +0000 UTC m=+1798.883062422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls") pod "maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" (UID: "42e3bc30-92ea-4442-acea-37c5fb3e371c") : secret "maas-api-e2e-mt-2cdc69a3-serving-cert" not found Jul 12 19:51:49.701093 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.701067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e3bc30-92ea-4442-acea-37c5fb3e371c-openshift-service-ca\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:49.711472 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:49.711444 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vcmd\" (UniqueName: \"kubernetes.io/projected/42e3bc30-92ea-4442-acea-37c5fb3e371c-kube-api-access-7vcmd\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:50.206979 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:50.206940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:50.209409 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:50.209380 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls\") pod \"maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:50.464940 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:50.464905 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:50.604155 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:50.604049 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq"] Jul 12 19:51:50.606703 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:51:50.606671 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e3bc30_92ea_4442_acea_37c5fb3e371c.slice/crio-9f5ab42b90424106b487cb8edc07f41b3f0ab16fedd8d17fa7a827177bbfaa89 WatchSource:0}: Error finding container 9f5ab42b90424106b487cb8edc07f41b3f0ab16fedd8d17fa7a827177bbfaa89: Status 404 returned error can't find the container with id 9f5ab42b90424106b487cb8edc07f41b3f0ab16fedd8d17fa7a827177bbfaa89 Jul 12 19:51:51.011845 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:51.011810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" event={"ID":"42e3bc30-92ea-4442-acea-37c5fb3e371c","Type":"ContainerStarted","Data":"9f5ab42b90424106b487cb8edc07f41b3f0ab16fedd8d17fa7a827177bbfaa89"} Jul 12 19:51:51.891825 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:51.891787 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:51:51.893835 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:51.893813 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:51:51.895799 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:51.895778 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:51:51.897645 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:51.897628 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:51:52.017871 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:52.017830 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" event={"ID":"42e3bc30-92ea-4442-acea-37c5fb3e371c","Type":"ContainerStarted","Data":"dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801"} Jul 12 19:51:52.018042 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:52.017962 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:51:52.035033 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:52.034988 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" podStartSLOduration=2.699583413 podStartE2EDuration="3.0349727s" totalImagesPullingTime="335.389287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:51:52.0332427 +0000 UTC m=+1800.715721161" watchObservedRunningTime="2026-07-12 19:51:52.0349727 +0000 UTC m=+1800.717451160" Jul 12 19:51:55.870034 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:55.870003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:51:58.028198 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:51:58.028150 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:52:11.767783 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:11.767738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:52:36.065568 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:36.065526 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:52:41.897789 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:41.897757 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq"] Jul 12 19:52:41.898170 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:41.897971 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" podUID="42e3bc30-92ea-4442-acea-37c5fb3e371c" containerName="maas-api" containerID="cri-o://dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801" gracePeriod=30 Jul 12 19:52:42.152751 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.152693 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:52:42.260063 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.260025 2565 generic.go:356] "Generic (PLEG): container finished" podID="42e3bc30-92ea-4442-acea-37c5fb3e371c" containerID="dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801" exitCode=0 Jul 12 19:52:42.260280 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.260073 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" Jul 12 19:52:42.260280 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.260078 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" event={"ID":"42e3bc30-92ea-4442-acea-37c5fb3e371c","Type":"ContainerDied","Data":"dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801"} Jul 12 19:52:42.260280 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.260215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq" event={"ID":"42e3bc30-92ea-4442-acea-37c5fb3e371c","Type":"ContainerDied","Data":"9f5ab42b90424106b487cb8edc07f41b3f0ab16fedd8d17fa7a827177bbfaa89"} Jul 12 19:52:42.260280 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.260250 2565 scope.go:117] "RemoveContainer" containerID="dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801" Jul 12 19:52:42.268712 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.268696 2565 scope.go:117] "RemoveContainer" containerID="dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801" Jul 12 19:52:42.268948 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:52:42.268931 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801\": container with ID starting with dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801 not found: ID does not exist" containerID="dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801" Jul 12 19:52:42.268987 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.268957 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801"} err="failed to get container status \"dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801\": rpc error: code = NotFound desc = could not find container \"dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801\": container with ID starting with dba2186dab9b6a4d15ace798733906fd9b696b01f41bacd32ebe95642857e801 not found: ID does not exist" Jul 12 19:52:42.272217 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.272202 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vcmd\" (UniqueName: \"kubernetes.io/projected/42e3bc30-92ea-4442-acea-37c5fb3e371c-kube-api-access-7vcmd\") pod \"42e3bc30-92ea-4442-acea-37c5fb3e371c\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " Jul 12 19:52:42.272266 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.272257 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e3bc30-92ea-4442-acea-37c5fb3e371c-openshift-service-ca\") pod \"42e3bc30-92ea-4442-acea-37c5fb3e371c\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " Jul 12 19:52:42.272361 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.272343 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls\") pod \"42e3bc30-92ea-4442-acea-37c5fb3e371c\" (UID: \"42e3bc30-92ea-4442-acea-37c5fb3e371c\") " Jul 12 19:52:42.272587 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.272566 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e3bc30-92ea-4442-acea-37c5fb3e371c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42e3bc30-92ea-4442-acea-37c5fb3e371c" (UID: "42e3bc30-92ea-4442-acea-37c5fb3e371c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:52:42.274389 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.274368 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42e3bc30-92ea-4442-acea-37c5fb3e371c" (UID: "42e3bc30-92ea-4442-acea-37c5fb3e371c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:52:42.274485 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.274451 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42e3bc30-92ea-4442-acea-37c5fb3e371c-kube-api-access-7vcmd" (OuterVolumeSpecName: "kube-api-access-7vcmd") pod "42e3bc30-92ea-4442-acea-37c5fb3e371c" (UID: "42e3bc30-92ea-4442-acea-37c5fb3e371c"). InnerVolumeSpecName "kube-api-access-7vcmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:52:42.373764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.373732 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42e3bc30-92ea-4442-acea-37c5fb3e371c-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:52:42.373764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.373760 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vcmd\" (UniqueName: \"kubernetes.io/projected/42e3bc30-92ea-4442-acea-37c5fb3e371c-kube-api-access-7vcmd\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:52:42.373764 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.373770 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e3bc30-92ea-4442-acea-37c5fb3e371c-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:52:42.581448 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.581407 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq"] Jul 12 19:52:42.583642 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:42.583610 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-mt-2cdc69a3-68987f8745-4pvcq"] Jul 12 19:52:43.862054 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:43.862005 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e3bc30-92ea-4442-acea-37c5fb3e371c" path="/var/lib/kubelet/pods/42e3bc30-92ea-4442-acea-37c5fb3e371c/volumes" Jul 12 19:52:46.072506 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:46.072467 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:52:55.468446 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:52:55.468405 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:53:40.673347 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:53:40.673313 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:54:01.771118 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:01.771082 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:54:15.571059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:15.571022 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:54:54.248516 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.248425 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l"] Jul 12 19:54:54.249076 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.248803 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e3bc30-92ea-4442-acea-37c5fb3e371c" containerName="maas-api" Jul 12 19:54:54.249076 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.248816 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e3bc30-92ea-4442-acea-37c5fb3e371c" containerName="maas-api" Jul 12 19:54:54.249076 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.248905 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e3bc30-92ea-4442-acea-37c5fb3e371c" containerName="maas-api" Jul 12 19:54:54.252394 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.252371 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.254672 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.254650 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-a-515e44-serving-cert\"" Jul 12 19:54:54.264734 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.264710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l"] Jul 12 19:54:54.318441 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.318405 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7tb\" (UniqueName: \"kubernetes.io/projected/e5677847-1124-4907-8f1c-76ce83c7635e-kube-api-access-jd7tb\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.318610 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.318455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5677847-1124-4907-8f1c-76ce83c7635e-maas-api-tls\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.318610 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.318478 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5677847-1124-4907-8f1c-76ce83c7635e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.419048 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.419011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7tb\" (UniqueName: \"kubernetes.io/projected/e5677847-1124-4907-8f1c-76ce83c7635e-kube-api-access-jd7tb\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.419258 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.419074 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5677847-1124-4907-8f1c-76ce83c7635e-maas-api-tls\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.419258 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.419112 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5677847-1124-4907-8f1c-76ce83c7635e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.419794 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.419772 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5677847-1124-4907-8f1c-76ce83c7635e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.421454 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.421433 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5677847-1124-4907-8f1c-76ce83c7635e-maas-api-tls\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.426413 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.426389 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7tb\" (UniqueName: \"kubernetes.io/projected/e5677847-1124-4907-8f1c-76ce83c7635e-kube-api-access-jd7tb\") pod \"maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.563705 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.563624 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:54.699109 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.699050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l"] Jul 12 19:54:54.701605 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:54:54.701573 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5677847_1124_4907_8f1c_76ce83c7635e.slice/crio-760f53c7fe832b9e03b43a9f1da254fa73aebc0c899d3e559b49dc95fb0ff102 WatchSource:0}: Error finding container 760f53c7fe832b9e03b43a9f1da254fa73aebc0c899d3e559b49dc95fb0ff102: Status 404 returned error can't find the container with id 760f53c7fe832b9e03b43a9f1da254fa73aebc0c899d3e559b49dc95fb0ff102 Jul 12 19:54:54.880045 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:54.880010 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" event={"ID":"e5677847-1124-4907-8f1c-76ce83c7635e","Type":"ContainerStarted","Data":"760f53c7fe832b9e03b43a9f1da254fa73aebc0c899d3e559b49dc95fb0ff102"} Jul 12 19:54:55.886069 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:55.886030 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" event={"ID":"e5677847-1124-4907-8f1c-76ce83c7635e","Type":"ContainerStarted","Data":"5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41"} Jul 12 19:54:55.886542 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:55.886235 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:54:55.903017 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:54:55.902974 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" podStartSLOduration=1.522693269 podStartE2EDuration="1.902957243s" totalImagesPullingTime="380.263974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:54:55.900947993 +0000 UTC m=+1984.583426454" watchObservedRunningTime="2026-07-12 19:54:55.902957243 +0000 UTC m=+1984.585435704" Jul 12 19:55:01.473664 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:01.473629 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:55:01.896622 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:01.896586 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:55:02.307860 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.307772 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj"] Jul 12 19:55:02.311817 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.311790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.313964 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.313932 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-a78ace-serving-cert\"" Jul 12 19:55:02.321400 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.321366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj"] Jul 12 19:55:02.387545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.387512 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.387545 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.387547 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjs89\" (UniqueName: \"kubernetes.io/projected/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-kube-api-access-xjs89\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.387756 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.387623 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.488710 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.488669 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.489072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.488716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjs89\" (UniqueName: \"kubernetes.io/projected/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-kube-api-access-xjs89\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.489072 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.488774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.489072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:55:02.488944 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-a78ace-serving-cert: secret "maas-api-e2e-inf-b-a78ace-serving-cert" not found Jul 12 19:55:02.489072 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:55:02.489004 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls podName:2d4cdd4d-e10b-4899-bfa2-52a1209650ff nodeName:}" failed. No retries permitted until 2026-07-12 19:55:02.988983848 +0000 UTC m=+1991.671462290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls") pod "maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" (UID: "2d4cdd4d-e10b-4899-bfa2-52a1209650ff") : secret "maas-api-e2e-inf-b-a78ace-serving-cert" not found Jul 12 19:55:02.489450 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.489429 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.497170 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.497146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjs89\" (UniqueName: \"kubernetes.io/projected/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-kube-api-access-xjs89\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.993308 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.993274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:02.995874 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:02.995844 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls\") pod \"maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:03.227623 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:03.227577 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:03.366284 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:03.366257 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj"] Jul 12 19:55:03.368588 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:55:03.368558 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4cdd4d_e10b_4899_bfa2_52a1209650ff.slice/crio-2128f9fe523d5f469c78ce53ecc1e975b0afe34e7edf08ba6a9945edd8bc9394 WatchSource:0}: Error finding container 2128f9fe523d5f469c78ce53ecc1e975b0afe34e7edf08ba6a9945edd8bc9394: Status 404 returned error can't find the container with id 2128f9fe523d5f469c78ce53ecc1e975b0afe34e7edf08ba6a9945edd8bc9394 Jul 12 19:55:03.927172 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:03.927137 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" event={"ID":"2d4cdd4d-e10b-4899-bfa2-52a1209650ff","Type":"ContainerStarted","Data":"063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1"} Jul 12 19:55:03.927604 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:03.927209 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" event={"ID":"2d4cdd4d-e10b-4899-bfa2-52a1209650ff","Type":"ContainerStarted","Data":"2128f9fe523d5f469c78ce53ecc1e975b0afe34e7edf08ba6a9945edd8bc9394"} Jul 12 19:55:03.927604 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:03.927290 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:03.942622 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:03.942556 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" podStartSLOduration=1.597318413 podStartE2EDuration="1.942539032s" totalImagesPullingTime="345.220619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:55:03.941431141 +0000 UTC m=+1992.623909605" watchObservedRunningTime="2026-07-12 19:55:03.942539032 +0000 UTC m=+1992.625017553" Jul 12 19:55:09.938986 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:09.938950 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:55:13.478302 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:13.478264 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:55:30.578244 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:30.578210 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:55:48.676898 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:55:48.676850 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:56:01.972520 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:01.972484 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:56:06.794361 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:06.794324 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l"] Jul 12 19:56:06.794783 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:06.794567 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" podUID="e5677847-1124-4907-8f1c-76ce83c7635e" containerName="maas-api" containerID="cri-o://5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41" gracePeriod=30 Jul 12 19:56:06.891898 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:06.891854 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" podUID="e5677847-1124-4907-8f1c-76ce83c7635e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.45:8443/health\": dial tcp 10.132.0.45:8443: connect: connection refused" Jul 12 19:56:07.046698 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.046632 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:56:07.165059 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.165025 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5677847-1124-4907-8f1c-76ce83c7635e-openshift-service-ca\") pod \"e5677847-1124-4907-8f1c-76ce83c7635e\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " Jul 12 19:56:07.165288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.165089 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5677847-1124-4907-8f1c-76ce83c7635e-maas-api-tls\") pod \"e5677847-1124-4907-8f1c-76ce83c7635e\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " Jul 12 19:56:07.165288 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.165111 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jd7tb\" (UniqueName: \"kubernetes.io/projected/e5677847-1124-4907-8f1c-76ce83c7635e-kube-api-access-jd7tb\") pod \"e5677847-1124-4907-8f1c-76ce83c7635e\" (UID: \"e5677847-1124-4907-8f1c-76ce83c7635e\") " Jul 12 19:56:07.165492 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.165457 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5677847-1124-4907-8f1c-76ce83c7635e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5677847-1124-4907-8f1c-76ce83c7635e" (UID: "e5677847-1124-4907-8f1c-76ce83c7635e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:56:07.167276 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.167242 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5677847-1124-4907-8f1c-76ce83c7635e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5677847-1124-4907-8f1c-76ce83c7635e" (UID: "e5677847-1124-4907-8f1c-76ce83c7635e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:56:07.167360 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.167302 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5677847-1124-4907-8f1c-76ce83c7635e-kube-api-access-jd7tb" (OuterVolumeSpecName: "kube-api-access-jd7tb") pod "e5677847-1124-4907-8f1c-76ce83c7635e" (UID: "e5677847-1124-4907-8f1c-76ce83c7635e"). InnerVolumeSpecName "kube-api-access-jd7tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:07.231519 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.231482 2565 generic.go:356] "Generic (PLEG): container finished" podID="e5677847-1124-4907-8f1c-76ce83c7635e" containerID="5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41" exitCode=0 Jul 12 19:56:07.231710 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.231546 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" Jul 12 19:56:07.231710 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.231576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" event={"ID":"e5677847-1124-4907-8f1c-76ce83c7635e","Type":"ContainerDied","Data":"5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41"} Jul 12 19:56:07.231710 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.231663 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l" event={"ID":"e5677847-1124-4907-8f1c-76ce83c7635e","Type":"ContainerDied","Data":"760f53c7fe832b9e03b43a9f1da254fa73aebc0c899d3e559b49dc95fb0ff102"} Jul 12 19:56:07.231710 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.231693 2565 scope.go:117] "RemoveContainer" containerID="5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41" Jul 12 19:56:07.241431 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.241416 2565 scope.go:117] "RemoveContainer" containerID="5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41" Jul 12 19:56:07.241668 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:56:07.241652 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41\": container with ID starting with 5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41 not found: ID does not exist" containerID="5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41" Jul 12 19:56:07.241707 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.241676 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41"} err="failed to get container status \"5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41\": rpc error: code = NotFound desc = could not find container \"5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41\": container with ID starting with 5a697458b92d69cad3ecec045af1658a6530f46c20d306d52ef008cde7919a41 not found: ID does not exist" Jul 12 19:56:07.252937 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.252916 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l"] Jul 12 19:56:07.258275 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.258252 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-a-515e44-6c6d7cf84d-5zp7l"] Jul 12 19:56:07.266633 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.266615 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5677847-1124-4907-8f1c-76ce83c7635e-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:56:07.266716 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.266635 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5677847-1124-4907-8f1c-76ce83c7635e-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:56:07.266716 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.266645 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd7tb\" (UniqueName: \"kubernetes.io/projected/e5677847-1124-4907-8f1c-76ce83c7635e-kube-api-access-jd7tb\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:56:07.862070 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:07.862026 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5677847-1124-4907-8f1c-76ce83c7635e" path="/var/lib/kubelet/pods/e5677847-1124-4907-8f1c-76ce83c7635e/volumes" Jul 12 19:56:17.973080 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:17.973042 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:56:49.640676 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:49.640638 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj"] Jul 12 19:56:49.641520 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:49.640871 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" podUID="2d4cdd4d-e10b-4899-bfa2-52a1209650ff" containerName="maas-api" containerID="cri-o://063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1" gracePeriod=30 Jul 12 19:56:49.898418 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:49.898356 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:56:50.046583 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.046540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls\") pod \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " Jul 12 19:56:50.046785 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.046642 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-openshift-service-ca\") pod \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " Jul 12 19:56:50.046785 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.046726 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjs89\" (UniqueName: \"kubernetes.io/projected/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-kube-api-access-xjs89\") pod \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\" (UID: \"2d4cdd4d-e10b-4899-bfa2-52a1209650ff\") " Jul 12 19:56:50.047041 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.047006 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2d4cdd4d-e10b-4899-bfa2-52a1209650ff" (UID: "2d4cdd4d-e10b-4899-bfa2-52a1209650ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:56:50.048753 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.048719 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2d4cdd4d-e10b-4899-bfa2-52a1209650ff" (UID: "2d4cdd4d-e10b-4899-bfa2-52a1209650ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:56:50.048931 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.048906 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-kube-api-access-xjs89" (OuterVolumeSpecName: "kube-api-access-xjs89") pod "2d4cdd4d-e10b-4899-bfa2-52a1209650ff" (UID: "2d4cdd4d-e10b-4899-bfa2-52a1209650ff"). InnerVolumeSpecName "kube-api-access-xjs89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:56:50.148125 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.148089 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.148125 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.148118 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.148125 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.148132 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjs89\" (UniqueName: \"kubernetes.io/projected/2d4cdd4d-e10b-4899-bfa2-52a1209650ff-kube-api-access-xjs89\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:56:50.443304 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.443267 2565 generic.go:356] "Generic (PLEG): container finished" podID="2d4cdd4d-e10b-4899-bfa2-52a1209650ff" containerID="063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1" exitCode=0 Jul 12 19:56:50.443480 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.443328 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" Jul 12 19:56:50.443480 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.443352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" event={"ID":"2d4cdd4d-e10b-4899-bfa2-52a1209650ff","Type":"ContainerDied","Data":"063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1"} Jul 12 19:56:50.443480 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.443426 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj" event={"ID":"2d4cdd4d-e10b-4899-bfa2-52a1209650ff","Type":"ContainerDied","Data":"2128f9fe523d5f469c78ce53ecc1e975b0afe34e7edf08ba6a9945edd8bc9394"} Jul 12 19:56:50.443480 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.443448 2565 scope.go:117] "RemoveContainer" containerID="063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1" Jul 12 19:56:50.452357 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.452343 2565 scope.go:117] "RemoveContainer" containerID="063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1" Jul 12 19:56:50.452611 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:56:50.452592 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1\": container with ID starting with 063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1 not found: ID does not exist" containerID="063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1" Jul 12 19:56:50.452695 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.452621 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1"} err="failed to get container status \"063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1\": rpc error: code = NotFound desc = could not find container \"063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1\": container with ID starting with 063acf1de86c112d8b8d2ca330316722f2a8e1c375a08f29cb1c43383e37dbb1 not found: ID does not exist" Jul 12 19:56:50.463414 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.463390 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj"] Jul 12 19:56:50.468932 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:50.468910 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-a78ace-5f7bb5fb-vhvxj"] Jul 12 19:56:51.862237 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:51.862208 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d4cdd4d-e10b-4899-bfa2-52a1209650ff" path="/var/lib/kubelet/pods/2d4cdd4d-e10b-4899-bfa2-52a1209650ff/volumes" Jul 12 19:56:51.921213 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:51.921164 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:56:51.923488 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:51.923462 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 19:56:51.925096 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:51.925075 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:56:51.927793 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:51.927773 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 19:56:52.974267 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:52.974236 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:56:58.800212 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.800151 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh"] Jul 12 19:56:58.801289 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.801260 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d4cdd4d-e10b-4899-bfa2-52a1209650ff" containerName="maas-api" Jul 12 19:56:58.801289 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.801289 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d4cdd4d-e10b-4899-bfa2-52a1209650ff" containerName="maas-api" Jul 12 19:56:58.801439 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.801323 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5677847-1124-4907-8f1c-76ce83c7635e" containerName="maas-api" Jul 12 19:56:58.801439 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.801332 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5677847-1124-4907-8f1c-76ce83c7635e" containerName="maas-api" Jul 12 19:56:58.801510 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.801441 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d4cdd4d-e10b-4899-bfa2-52a1209650ff" containerName="maas-api" Jul 12 19:56:58.801510 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.801455 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5677847-1124-4907-8f1c-76ce83c7635e" containerName="maas-api" Jul 12 19:56:58.804856 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.804837 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:58.807404 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.807382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-b032b1-serving-cert\"" Jul 12 19:56:58.810844 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.810821 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh"] Jul 12 19:56:58.923437 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.923402 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a25304-f098-4e6d-a5d3-a1eb745ca226-openshift-service-ca\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:58.923651 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.923459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946qg\" (UniqueName: \"kubernetes.io/projected/b9a25304-f098-4e6d-a5d3-a1eb745ca226-kube-api-access-946qg\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:58.923651 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:58.923520 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.024652 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.024617 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946qg\" (UniqueName: \"kubernetes.io/projected/b9a25304-f098-4e6d-a5d3-a1eb745ca226-kube-api-access-946qg\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.024974 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.024941 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.025120 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.025103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a25304-f098-4e6d-a5d3-a1eb745ca226-openshift-service-ca\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.025222 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:56:59.025108 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-b032b1-serving-cert: secret "maas-api-e2e-api-a-b032b1-serving-cert" not found Jul 12 19:56:59.025222 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:56:59.025221 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls podName:b9a25304-f098-4e6d-a5d3-a1eb745ca226 nodeName:}" failed. No retries permitted until 2026-07-12 19:56:59.525203334 +0000 UTC m=+2108.207681786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls") pod "maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" (UID: "b9a25304-f098-4e6d-a5d3-a1eb745ca226") : secret "maas-api-e2e-api-a-b032b1-serving-cert" not found Jul 12 19:56:59.025765 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.025745 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a25304-f098-4e6d-a5d3-a1eb745ca226-openshift-service-ca\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.033556 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.033537 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-946qg\" (UniqueName: \"kubernetes.io/projected/b9a25304-f098-4e6d-a5d3-a1eb745ca226-kube-api-access-946qg\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.529447 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.529408 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.531837 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.531801 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls\") pod \"maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.718815 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.718778 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:56:59.860654 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:56:59.860624 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a25304_f098_4e6d_a5d3_a1eb745ca226.slice/crio-e0d84d755be2bc4487c6edc216704fae4c1d2337c59aeedb111f2ff7f6bd43d3 WatchSource:0}: Error finding container e0d84d755be2bc4487c6edc216704fae4c1d2337c59aeedb111f2ff7f6bd43d3: Status 404 returned error can't find the container with id e0d84d755be2bc4487c6edc216704fae4c1d2337c59aeedb111f2ff7f6bd43d3 Jul 12 19:56:59.862029 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.862006 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 12 19:56:59.862557 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:56:59.862535 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh"] Jul 12 19:57:00.493601 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:00.493555 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" event={"ID":"b9a25304-f098-4e6d-a5d3-a1eb745ca226","Type":"ContainerStarted","Data":"f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1"} Jul 12 19:57:00.493770 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:00.493619 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" event={"ID":"b9a25304-f098-4e6d-a5d3-a1eb745ca226","Type":"ContainerStarted","Data":"e0d84d755be2bc4487c6edc216704fae4c1d2337c59aeedb111f2ff7f6bd43d3"} Jul 12 19:57:00.493770 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:00.493646 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:57:00.509320 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:00.509226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" podStartSLOduration=2.113372527 podStartE2EDuration="2.509210575s" totalImagesPullingTime="395.838048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:56:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:00.50779034 +0000 UTC m=+2109.190268802" watchObservedRunningTime="2026-07-12 19:57:00.509210575 +0000 UTC m=+2109.191689037" Jul 12 19:57:02.568078 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:02.568034 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:57:06.503915 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:06.503880 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:57:07.127484 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.127453 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf"] Jul 12 19:57:07.131131 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.131113 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.133461 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.133433 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-b-202584-serving-cert\"" Jul 12 19:57:07.140846 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.140820 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf"] Jul 12 19:57:07.301098 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.301056 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a833dc-775b-4f3f-abbd-10e0edf53c11-openshift-service-ca\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.301291 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.301125 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a833dc-775b-4f3f-abbd-10e0edf53c11-maas-api-tls\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.301291 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.301198 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lncdj\" (UniqueName: \"kubernetes.io/projected/e1a833dc-775b-4f3f-abbd-10e0edf53c11-kube-api-access-lncdj\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.402329 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.402234 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a833dc-775b-4f3f-abbd-10e0edf53c11-openshift-service-ca\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.402329 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.402302 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a833dc-775b-4f3f-abbd-10e0edf53c11-maas-api-tls\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.402579 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.402339 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lncdj\" (UniqueName: \"kubernetes.io/projected/e1a833dc-775b-4f3f-abbd-10e0edf53c11-kube-api-access-lncdj\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.402992 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.402964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a833dc-775b-4f3f-abbd-10e0edf53c11-openshift-service-ca\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.404891 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.404867 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a833dc-775b-4f3f-abbd-10e0edf53c11-maas-api-tls\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.411664 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.411645 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lncdj\" (UniqueName: \"kubernetes.io/projected/e1a833dc-775b-4f3f-abbd-10e0edf53c11-kube-api-access-lncdj\") pod \"maas-api-e2e-api-b-202584-68b4b75859-lqwlf\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.444682 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.444637 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:07.604784 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:07.604762 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf"] Jul 12 19:57:07.607077 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:57:07.607050 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a833dc_775b_4f3f_abbd_10e0edf53c11.slice/crio-6aee1c16622af2535d2cc56aac4042f5fb2bec4ffa0509f8d7f7a4e667960d32 WatchSource:0}: Error finding container 6aee1c16622af2535d2cc56aac4042f5fb2bec4ffa0509f8d7f7a4e667960d32: Status 404 returned error can't find the container with id 6aee1c16622af2535d2cc56aac4042f5fb2bec4ffa0509f8d7f7a4e667960d32 Jul 12 19:57:08.537403 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:08.537369 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" event={"ID":"e1a833dc-775b-4f3f-abbd-10e0edf53c11","Type":"ContainerStarted","Data":"831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2"} Jul 12 19:57:08.537629 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:08.537426 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:08.537629 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:08.537442 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" event={"ID":"e1a833dc-775b-4f3f-abbd-10e0edf53c11","Type":"ContainerStarted","Data":"6aee1c16622af2535d2cc56aac4042f5fb2bec4ffa0509f8d7f7a4e667960d32"} Jul 12 19:57:08.555541 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:08.555487 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" podStartSLOduration=1.171423821 podStartE2EDuration="1.55547021s" totalImagesPullingTime="384.046389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:08.553382398 +0000 UTC m=+2117.235860858" watchObservedRunningTime="2026-07-12 19:57:08.55547021 +0000 UTC m=+2117.237948714" Jul 12 19:57:14.548954 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:14.548924 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:16.978270 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:16.978232 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:57:20.110476 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.110436 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh"] Jul 12 19:57:20.111067 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.110684 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" podUID="b9a25304-f098-4e6d-a5d3-a1eb745ca226" containerName="maas-api" containerID="cri-o://f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1" gracePeriod=30 Jul 12 19:57:20.363530 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.363468 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:57:20.413639 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.413606 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls\") pod \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " Jul 12 19:57:20.413820 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.413648 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-946qg\" (UniqueName: \"kubernetes.io/projected/b9a25304-f098-4e6d-a5d3-a1eb745ca226-kube-api-access-946qg\") pod \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " Jul 12 19:57:20.413820 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.413674 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a25304-f098-4e6d-a5d3-a1eb745ca226-openshift-service-ca\") pod \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\" (UID: \"b9a25304-f098-4e6d-a5d3-a1eb745ca226\") " Jul 12 19:57:20.414052 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.414026 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9a25304-f098-4e6d-a5d3-a1eb745ca226-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b9a25304-f098-4e6d-a5d3-a1eb745ca226" (UID: "b9a25304-f098-4e6d-a5d3-a1eb745ca226"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:57:20.415827 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.415808 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a25304-f098-4e6d-a5d3-a1eb745ca226-kube-api-access-946qg" (OuterVolumeSpecName: "kube-api-access-946qg") pod "b9a25304-f098-4e6d-a5d3-a1eb745ca226" (UID: "b9a25304-f098-4e6d-a5d3-a1eb745ca226"). InnerVolumeSpecName "kube-api-access-946qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:20.415827 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.415815 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b9a25304-f098-4e6d-a5d3-a1eb745ca226" (UID: "b9a25304-f098-4e6d-a5d3-a1eb745ca226"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:57:20.514665 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.514629 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9a25304-f098-4e6d-a5d3-a1eb745ca226-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:57:20.514665 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.514659 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946qg\" (UniqueName: \"kubernetes.io/projected/b9a25304-f098-4e6d-a5d3-a1eb745ca226-kube-api-access-946qg\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:57:20.514665 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.514668 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9a25304-f098-4e6d-a5d3-a1eb745ca226-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:57:20.595300 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.595266 2565 generic.go:356] "Generic (PLEG): container finished" podID="b9a25304-f098-4e6d-a5d3-a1eb745ca226" containerID="f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1" exitCode=0 Jul 12 19:57:20.595467 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.595320 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" Jul 12 19:57:20.595467 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.595358 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" event={"ID":"b9a25304-f098-4e6d-a5d3-a1eb745ca226","Type":"ContainerDied","Data":"f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1"} Jul 12 19:57:20.595467 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.595417 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh" event={"ID":"b9a25304-f098-4e6d-a5d3-a1eb745ca226","Type":"ContainerDied","Data":"e0d84d755be2bc4487c6edc216704fae4c1d2337c59aeedb111f2ff7f6bd43d3"} Jul 12 19:57:20.595467 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.595446 2565 scope.go:117] "RemoveContainer" containerID="f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1" Jul 12 19:57:20.605672 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.605649 2565 scope.go:117] "RemoveContainer" containerID="f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1" Jul 12 19:57:20.605950 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:57:20.605929 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1\": container with ID starting with f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1 not found: ID does not exist" containerID="f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1" Jul 12 19:57:20.606015 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.605965 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1"} err="failed to get container status \"f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1\": rpc error: code = NotFound desc = could not find container \"f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1\": container with ID starting with f55cbbc4a4c69c560f76ebb8fc559cc344bbb504cd9ceab3dcaae11b05f54db1 not found: ID does not exist" Jul 12 19:57:20.616702 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.616643 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh"] Jul 12 19:57:20.621945 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:20.621918 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-b032b1-5db57786fc-pj8hh"] Jul 12 19:57:21.862387 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:21.862354 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a25304-f098-4e6d-a5d3-a1eb745ca226" path="/var/lib/kubelet/pods/b9a25304-f098-4e6d-a5d3-a1eb745ca226/volumes" Jul 12 19:57:26.804626 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:26.804588 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf"] Jul 12 19:57:26.805094 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:26.804844 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" podUID="e1a833dc-775b-4f3f-abbd-10e0edf53c11" containerName="maas-api" containerID="cri-o://831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2" gracePeriod=30 Jul 12 19:57:27.052832 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.052808 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:27.078980 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.078906 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lncdj\" (UniqueName: \"kubernetes.io/projected/e1a833dc-775b-4f3f-abbd-10e0edf53c11-kube-api-access-lncdj\") pod \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " Jul 12 19:57:27.078980 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.078954 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a833dc-775b-4f3f-abbd-10e0edf53c11-maas-api-tls\") pod \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " Jul 12 19:57:27.079223 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.079113 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a833dc-775b-4f3f-abbd-10e0edf53c11-openshift-service-ca\") pod \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\" (UID: \"e1a833dc-775b-4f3f-abbd-10e0edf53c11\") " Jul 12 19:57:27.079529 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.079499 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a833dc-775b-4f3f-abbd-10e0edf53c11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1a833dc-775b-4f3f-abbd-10e0edf53c11" (UID: "e1a833dc-775b-4f3f-abbd-10e0edf53c11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 19:57:27.081074 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.081044 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a833dc-775b-4f3f-abbd-10e0edf53c11-kube-api-access-lncdj" (OuterVolumeSpecName: "kube-api-access-lncdj") pod "e1a833dc-775b-4f3f-abbd-10e0edf53c11" (UID: "e1a833dc-775b-4f3f-abbd-10e0edf53c11"). InnerVolumeSpecName "kube-api-access-lncdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 19:57:27.081159 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.081067 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1a833dc-775b-4f3f-abbd-10e0edf53c11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1a833dc-775b-4f3f-abbd-10e0edf53c11" (UID: "e1a833dc-775b-4f3f-abbd-10e0edf53c11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 19:57:27.180447 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.180416 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1a833dc-775b-4f3f-abbd-10e0edf53c11-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:57:27.180447 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.180444 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lncdj\" (UniqueName: \"kubernetes.io/projected/e1a833dc-775b-4f3f-abbd-10e0edf53c11-kube-api-access-lncdj\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:57:27.180447 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.180454 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1a833dc-775b-4f3f-abbd-10e0edf53c11-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 19:57:27.629539 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.629507 2565 generic.go:356] "Generic (PLEG): container finished" podID="e1a833dc-775b-4f3f-abbd-10e0edf53c11" containerID="831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2" exitCode=0 Jul 12 19:57:27.629717 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.629551 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" Jul 12 19:57:27.629717 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.629600 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" event={"ID":"e1a833dc-775b-4f3f-abbd-10e0edf53c11","Type":"ContainerDied","Data":"831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2"} Jul 12 19:57:27.629717 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.629652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf" event={"ID":"e1a833dc-775b-4f3f-abbd-10e0edf53c11","Type":"ContainerDied","Data":"6aee1c16622af2535d2cc56aac4042f5fb2bec4ffa0509f8d7f7a4e667960d32"} Jul 12 19:57:27.629717 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.629677 2565 scope.go:117] "RemoveContainer" containerID="831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2" Jul 12 19:57:27.638659 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.638642 2565 scope.go:117] "RemoveContainer" containerID="831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2" Jul 12 19:57:27.638922 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:57:27.638905 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2\": container with ID starting with 831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2 not found: ID does not exist" containerID="831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2" Jul 12 19:57:27.638971 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.638931 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2"} err="failed to get container status \"831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2\": rpc error: code = NotFound desc = could not find container \"831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2\": container with ID starting with 831f8d9c7ae9b5ef8671788f28ccf87254ca76de6c59f1e03392183a7cf0b2a2 not found: ID does not exist" Jul 12 19:57:27.649159 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.649137 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf"] Jul 12 19:57:27.652761 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.652742 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-b-202584-68b4b75859-lqwlf"] Jul 12 19:57:27.862247 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:27.862206 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1a833dc-775b-4f3f-abbd-10e0edf53c11" path="/var/lib/kubelet/pods/e1a833dc-775b-4f3f-abbd-10e0edf53c11/volumes" Jul 12 19:57:32.676883 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:32.676842 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:57:35.975214 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.975135 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74"] Jul 12 19:57:35.975733 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.975714 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a833dc-775b-4f3f-abbd-10e0edf53c11" containerName="maas-api" Jul 12 19:57:35.975797 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.975736 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a833dc-775b-4f3f-abbd-10e0edf53c11" containerName="maas-api" Jul 12 19:57:35.975797 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.975759 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a25304-f098-4e6d-a5d3-a1eb745ca226" containerName="maas-api" Jul 12 19:57:35.975797 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.975767 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a25304-f098-4e6d-a5d3-a1eb745ca226" containerName="maas-api" Jul 12 19:57:35.975940 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.975870 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1a833dc-775b-4f3f-abbd-10e0edf53c11" containerName="maas-api" Jul 12 19:57:35.975940 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.975888 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a25304-f098-4e6d-a5d3-a1eb745ca226" containerName="maas-api" Jul 12 19:57:35.980559 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.980540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:35.982954 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.982932 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-a-e68438-serving-cert\"" Jul 12 19:57:35.992479 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:35.992445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74"] Jul 12 19:57:36.059253 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.059224 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.059444 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.059259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.059444 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.059278 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnwj\" (UniqueName: \"kubernetes.io/projected/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-kube-api-access-ddnwj\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.160422 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.160387 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.160608 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.160436 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.160608 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:57:36.160535 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-a-e68438-serving-cert: secret "maas-api-e2e-shared-a-e68438-serving-cert" not found Jul 12 19:57:36.160608 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.160551 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnwj\" (UniqueName: \"kubernetes.io/projected/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-kube-api-access-ddnwj\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.160608 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:57:36.160592 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls podName:9909c9bd-3cb6-4896-b1fb-3e657df3eee5 nodeName:}" failed. No retries permitted until 2026-07-12 19:57:36.660576468 +0000 UTC m=+2145.343054909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls") pod "maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" (UID: "9909c9bd-3cb6-4896-b1fb-3e657df3eee5") : secret "maas-api-e2e-shared-a-e68438-serving-cert" not found Jul 12 19:57:36.161084 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.161061 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.168592 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.168572 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnwj\" (UniqueName: \"kubernetes.io/projected/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-kube-api-access-ddnwj\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.666284 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.666248 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.668672 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.668649 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls\") pod \"maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:36.892717 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:36.892681 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:37.030247 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:37.030216 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74"] Jul 12 19:57:37.680242 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:37.680134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" event={"ID":"9909c9bd-3cb6-4896-b1fb-3e657df3eee5","Type":"ContainerStarted","Data":"a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361"} Jul 12 19:57:37.680242 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:37.680217 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:37.680242 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:37.680235 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" event={"ID":"9909c9bd-3cb6-4896-b1fb-3e657df3eee5","Type":"ContainerStarted","Data":"7d79a8852698377a5c7fdc028a52dcfc7cc67ba79b35270916d5c9fec9e9bb44"} Jul 12 19:57:37.695807 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:37.695753 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" podStartSLOduration=2.381001122 podStartE2EDuration="2.695738284s" totalImagesPullingTime="314.737162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:57:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:37.694424439 +0000 UTC m=+2146.376902899" watchObservedRunningTime="2026-07-12 19:57:37.695738284 +0000 UTC m=+2146.378216743" Jul 12 19:57:41.270396 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:41.270361 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:57:43.690933 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:43.690902 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 19:57:44.245123 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.245001 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx"] Jul 12 19:57:44.248670 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.248649 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.250870 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.250848 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-5865e5-serving-cert\"" Jul 12 19:57:44.256063 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.256039 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx"] Jul 12 19:57:44.332772 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.332735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.332772 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.332773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbtvh\" (UniqueName: \"kubernetes.io/projected/8bd8a707-00d1-4c76-a37d-590c37d37ac4-kube-api-access-mbtvh\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.332974 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.332838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd8a707-00d1-4c76-a37d-590c37d37ac4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.433426 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.433389 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd8a707-00d1-4c76-a37d-590c37d37ac4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.433617 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.433518 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.433617 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.433547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbtvh\" (UniqueName: \"kubernetes.io/projected/8bd8a707-00d1-4c76-a37d-590c37d37ac4-kube-api-access-mbtvh\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.433744 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:57:44.433655 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-5865e5-serving-cert: secret "maas-api-e2e-shared-b-5865e5-serving-cert" not found Jul 12 19:57:44.433744 ip-10-0-141-249 kubenswrapper[2565]: E0712 19:57:44.433714 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls podName:8bd8a707-00d1-4c76-a37d-590c37d37ac4 nodeName:}" failed. No retries permitted until 2026-07-12 19:57:44.933699123 +0000 UTC m=+2153.616177561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls") pod "maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" (UID: "8bd8a707-00d1-4c76-a37d-590c37d37ac4") : secret "maas-api-e2e-shared-b-5865e5-serving-cert" not found Jul 12 19:57:44.434039 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.434020 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd8a707-00d1-4c76-a37d-590c37d37ac4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.442496 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.442474 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbtvh\" (UniqueName: \"kubernetes.io/projected/8bd8a707-00d1-4c76-a37d-590c37d37ac4-kube-api-access-mbtvh\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.938527 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.938484 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:44.940833 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:44.940808 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls\") pod \"maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:45.162039 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:45.162010 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:45.299959 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:45.299936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx"] Jul 12 19:57:45.302566 ip-10-0-141-249 kubenswrapper[2565]: W0712 19:57:45.302509 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd8a707_00d1_4c76_a37d_590c37d37ac4.slice/crio-05f970a0be5c80527c424c9528813714b75c17cc71268677386c60149bebfe88 WatchSource:0}: Error finding container 05f970a0be5c80527c424c9528813714b75c17cc71268677386c60149bebfe88: Status 404 returned error can't find the container with id 05f970a0be5c80527c424c9528813714b75c17cc71268677386c60149bebfe88 Jul 12 19:57:45.723282 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:45.723254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" event={"ID":"8bd8a707-00d1-4c76-a37d-590c37d37ac4","Type":"ContainerStarted","Data":"05f970a0be5c80527c424c9528813714b75c17cc71268677386c60149bebfe88"} Jul 12 19:57:46.730230 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:46.730195 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" event={"ID":"8bd8a707-00d1-4c76-a37d-590c37d37ac4","Type":"ContainerStarted","Data":"2a30575504fb243a0a0fbe5b72a5b4615c2085f752de821631281d7ae9c7ba7d"} Jul 12 19:57:46.730595 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:46.730335 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:57:46.746225 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:46.746160 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" podStartSLOduration=2.340251478 podStartE2EDuration="2.746143361s" totalImagesPullingTime="405.891883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 19:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 19:57:46.745224858 +0000 UTC m=+2155.427703320" watchObservedRunningTime="2026-07-12 19:57:46.746143361 +0000 UTC m=+2155.428621847" Jul 12 19:57:49.767992 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:49.767956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:57:52.741361 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:57:52.741329 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 19:58:05.575912 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:58:05.575869 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:58:23.474517 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:58:23.474480 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:58:41.973747 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:58:41.973715 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:58:58.976412 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:58:58.976377 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:59:16.871113 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:59:16.871079 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:59:33.273588 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:59:33.273544 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 19:59:49.868850 ip-10-0-141-249 kubenswrapper[2565]: I0712 19:59:49.868814 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:00:00.130910 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.130859 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr"] Jul 12 20:00:00.134437 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.134367 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767"] Jul 12 20:00:00.134598 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.134469 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" Jul 12 20:00:00.137706 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.137685 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29731440-bjbtm"] Jul 12 20:00:00.137839 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.137739 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" Jul 12 20:00:00.141377 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.141355 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr"] Jul 12 20:00:00.141550 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.141487 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" Jul 12 20:00:00.143272 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.143221 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767"] Jul 12 20:00:00.146427 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.146404 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29731440-bjbtm"] Jul 12 20:00:00.290571 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.290535 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59j94\" (UniqueName: \"kubernetes.io/projected/c9e53efa-bf16-49d7-a1e0-78d5acfba367-kube-api-access-59j94\") pod \"maas-api-key-cleanup-29731440-bjbtm\" (UID: \"c9e53efa-bf16-49d7-a1e0-78d5acfba367\") " pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" Jul 12 20:00:00.290750 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.290594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssq2x\" (UniqueName: \"kubernetes.io/projected/3bea2f06-3c8b-4d55-aeea-91c8820f0c26-kube-api-access-ssq2x\") pod \"maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767\" (UID: \"3bea2f06-3c8b-4d55-aeea-91c8820f0c26\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" Jul 12 20:00:00.290750 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.290659 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776kr\" (UniqueName: \"kubernetes.io/projected/0c42829b-9efc-43b1-bbc8-21617c31596d-kube-api-access-776kr\") pod \"maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr\" (UID: \"0c42829b-9efc-43b1-bbc8-21617c31596d\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" Jul 12 20:00:00.391404 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.391312 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59j94\" (UniqueName: \"kubernetes.io/projected/c9e53efa-bf16-49d7-a1e0-78d5acfba367-kube-api-access-59j94\") pod \"maas-api-key-cleanup-29731440-bjbtm\" (UID: \"c9e53efa-bf16-49d7-a1e0-78d5acfba367\") " pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" Jul 12 20:00:00.391404 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.391394 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssq2x\" (UniqueName: \"kubernetes.io/projected/3bea2f06-3c8b-4d55-aeea-91c8820f0c26-kube-api-access-ssq2x\") pod \"maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767\" (UID: \"3bea2f06-3c8b-4d55-aeea-91c8820f0c26\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" Jul 12 20:00:00.391611 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.391421 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776kr\" (UniqueName: \"kubernetes.io/projected/0c42829b-9efc-43b1-bbc8-21617c31596d-kube-api-access-776kr\") pod \"maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr\" (UID: \"0c42829b-9efc-43b1-bbc8-21617c31596d\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" Jul 12 20:00:00.399535 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.399510 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59j94\" (UniqueName: \"kubernetes.io/projected/c9e53efa-bf16-49d7-a1e0-78d5acfba367-kube-api-access-59j94\") pod \"maas-api-key-cleanup-29731440-bjbtm\" (UID: \"c9e53efa-bf16-49d7-a1e0-78d5acfba367\") " pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" Jul 12 20:00:00.400084 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.400063 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssq2x\" (UniqueName: \"kubernetes.io/projected/3bea2f06-3c8b-4d55-aeea-91c8820f0c26-kube-api-access-ssq2x\") pod \"maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767\" (UID: \"3bea2f06-3c8b-4d55-aeea-91c8820f0c26\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" Jul 12 20:00:00.400451 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.400430 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-776kr\" (UniqueName: \"kubernetes.io/projected/0c42829b-9efc-43b1-bbc8-21617c31596d-kube-api-access-776kr\") pod \"maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr\" (UID: \"0c42829b-9efc-43b1-bbc8-21617c31596d\") " pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" Jul 12 20:00:00.449140 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.449105 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" Jul 12 20:00:00.458165 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.458143 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" Jul 12 20:00:00.465055 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.465008 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" Jul 12 20:00:00.813263 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.813235 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr"] Jul 12 20:00:00.815562 ip-10-0-141-249 kubenswrapper[2565]: W0712 20:00:00.815533 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c42829b_9efc_43b1_bbc8_21617c31596d.slice/crio-063fb1c92ceab89c5bcc55d473627403825d70bebce3f06f7772a70434c326e6 WatchSource:0}: Error finding container 063fb1c92ceab89c5bcc55d473627403825d70bebce3f06f7772a70434c326e6: Status 404 returned error can't find the container with id 063fb1c92ceab89c5bcc55d473627403825d70bebce3f06f7772a70434c326e6 Jul 12 20:00:00.844978 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.844956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767"] Jul 12 20:00:00.847732 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:00.847713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29731440-bjbtm"] Jul 12 20:00:00.849427 ip-10-0-141-249 kubenswrapper[2565]: W0712 20:00:00.849403 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bea2f06_3c8b_4d55_aeea_91c8820f0c26.slice/crio-40e6aefa1e9383d0571046185121f98491e7a5a3517a928d7d3292b465d704fb WatchSource:0}: Error finding container 40e6aefa1e9383d0571046185121f98491e7a5a3517a928d7d3292b465d704fb: Status 404 returned error can't find the container with id 40e6aefa1e9383d0571046185121f98491e7a5a3517a928d7d3292b465d704fb Jul 12 20:00:00.849705 ip-10-0-141-249 kubenswrapper[2565]: W0712 20:00:00.849684 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e53efa_bf16_49d7_a1e0_78d5acfba367.slice/crio-822526ce9b79abd9c607ddbc56e91900ca5a76170211a83bd0837a43044bff91 WatchSource:0}: Error finding container 822526ce9b79abd9c607ddbc56e91900ca5a76170211a83bd0837a43044bff91: Status 404 returned error can't find the container with id 822526ce9b79abd9c607ddbc56e91900ca5a76170211a83bd0837a43044bff91 Jul 12 20:00:01.385417 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.385377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" event={"ID":"3bea2f06-3c8b-4d55-aeea-91c8820f0c26","Type":"ContainerStarted","Data":"1e23046291e03b81bb1b304939711bb89ee5f640f4ee8a233a6b567b62ae6d88"} Jul 12 20:00:01.385860 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.385471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" event={"ID":"3bea2f06-3c8b-4d55-aeea-91c8820f0c26","Type":"ContainerStarted","Data":"40e6aefa1e9383d0571046185121f98491e7a5a3517a928d7d3292b465d704fb"} Jul 12 20:00:01.387083 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.387055 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" event={"ID":"0c42829b-9efc-43b1-bbc8-21617c31596d","Type":"ContainerStarted","Data":"0e1dcd86dcb11405a00343466822a0a6cf2fcdd607d2ef4e5cff77c8d8b3d3e6"} Jul 12 20:00:01.387212 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.387096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" event={"ID":"0c42829b-9efc-43b1-bbc8-21617c31596d","Type":"ContainerStarted","Data":"063fb1c92ceab89c5bcc55d473627403825d70bebce3f06f7772a70434c326e6"} Jul 12 20:00:01.388638 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.388615 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" event={"ID":"c9e53efa-bf16-49d7-a1e0-78d5acfba367","Type":"ContainerStarted","Data":"ae309bf4c3054419ea983f7abb5f4bd4bfb79dd51aac22dbd05571a6a2dad922"} Jul 12 20:00:01.388751 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.388652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" event={"ID":"c9e53efa-bf16-49d7-a1e0-78d5acfba367","Type":"ContainerStarted","Data":"822526ce9b79abd9c607ddbc56e91900ca5a76170211a83bd0837a43044bff91"} Jul 12 20:00:01.401046 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.401005 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" podStartSLOduration=1.400992326 podStartE2EDuration="1.400992326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:01.397895846 +0000 UTC m=+2290.080374307" watchObservedRunningTime="2026-07-12 20:00:01.400992326 +0000 UTC m=+2290.083470785" Jul 12 20:00:01.411109 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.411072 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" podStartSLOduration=1.411060584 podStartE2EDuration="1.411060584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:01.409708274 +0000 UTC m=+2290.092186737" watchObservedRunningTime="2026-07-12 20:00:01.411060584 +0000 UTC m=+2290.093539043" Jul 12 20:00:01.423831 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:01.423793 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" podStartSLOduration=1.423782075 podStartE2EDuration="1.423782075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:01.42160143 +0000 UTC m=+2290.104079894" watchObservedRunningTime="2026-07-12 20:00:01.423782075 +0000 UTC m=+2290.106260535" Jul 12 20:00:02.396472 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:02.396432 2565 generic.go:356] "Generic (PLEG): container finished" podID="3bea2f06-3c8b-4d55-aeea-91c8820f0c26" containerID="1e23046291e03b81bb1b304939711bb89ee5f640f4ee8a233a6b567b62ae6d88" exitCode=0 Jul 12 20:00:02.396945 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:02.396482 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" event={"ID":"3bea2f06-3c8b-4d55-aeea-91c8820f0c26","Type":"ContainerDied","Data":"1e23046291e03b81bb1b304939711bb89ee5f640f4ee8a233a6b567b62ae6d88"} Jul 12 20:00:02.397998 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:02.397971 2565 generic.go:356] "Generic (PLEG): container finished" podID="0c42829b-9efc-43b1-bbc8-21617c31596d" containerID="0e1dcd86dcb11405a00343466822a0a6cf2fcdd607d2ef4e5cff77c8d8b3d3e6" exitCode=0 Jul 12 20:00:02.398112 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:02.398043 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" event={"ID":"0c42829b-9efc-43b1-bbc8-21617c31596d","Type":"ContainerDied","Data":"0e1dcd86dcb11405a00343466822a0a6cf2fcdd607d2ef4e5cff77c8d8b3d3e6"} Jul 12 20:00:02.399444 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:02.399417 2565 generic.go:356] "Generic (PLEG): container finished" podID="c9e53efa-bf16-49d7-a1e0-78d5acfba367" containerID="ae309bf4c3054419ea983f7abb5f4bd4bfb79dd51aac22dbd05571a6a2dad922" exitCode=0 Jul 12 20:00:02.399544 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:02.399447 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" event={"ID":"c9e53efa-bf16-49d7-a1e0-78d5acfba367","Type":"ContainerDied","Data":"ae309bf4c3054419ea983f7abb5f4bd4bfb79dd51aac22dbd05571a6a2dad922"} Jul 12 20:00:03.570150 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.570122 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" Jul 12 20:00:03.572925 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.572897 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" Jul 12 20:00:03.610953 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.610928 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" Jul 12 20:00:03.719734 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.719648 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssq2x\" (UniqueName: \"kubernetes.io/projected/3bea2f06-3c8b-4d55-aeea-91c8820f0c26-kube-api-access-ssq2x\") pod \"3bea2f06-3c8b-4d55-aeea-91c8820f0c26\" (UID: \"3bea2f06-3c8b-4d55-aeea-91c8820f0c26\") " Jul 12 20:00:03.719881 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.719742 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-776kr\" (UniqueName: \"kubernetes.io/projected/0c42829b-9efc-43b1-bbc8-21617c31596d-kube-api-access-776kr\") pod \"0c42829b-9efc-43b1-bbc8-21617c31596d\" (UID: \"0c42829b-9efc-43b1-bbc8-21617c31596d\") " Jul 12 20:00:03.719881 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.719780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59j94\" (UniqueName: \"kubernetes.io/projected/c9e53efa-bf16-49d7-a1e0-78d5acfba367-kube-api-access-59j94\") pod \"c9e53efa-bf16-49d7-a1e0-78d5acfba367\" (UID: \"c9e53efa-bf16-49d7-a1e0-78d5acfba367\") " Jul 12 20:00:03.721912 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.721879 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e53efa-bf16-49d7-a1e0-78d5acfba367-kube-api-access-59j94" (OuterVolumeSpecName: "kube-api-access-59j94") pod "c9e53efa-bf16-49d7-a1e0-78d5acfba367" (UID: "c9e53efa-bf16-49d7-a1e0-78d5acfba367"). InnerVolumeSpecName "kube-api-access-59j94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:00:03.721912 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.721900 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c42829b-9efc-43b1-bbc8-21617c31596d-kube-api-access-776kr" (OuterVolumeSpecName: "kube-api-access-776kr") pod "0c42829b-9efc-43b1-bbc8-21617c31596d" (UID: "0c42829b-9efc-43b1-bbc8-21617c31596d"). InnerVolumeSpecName "kube-api-access-776kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:00:03.722083 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.721933 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bea2f06-3c8b-4d55-aeea-91c8820f0c26-kube-api-access-ssq2x" (OuterVolumeSpecName: "kube-api-access-ssq2x") pod "3bea2f06-3c8b-4d55-aeea-91c8820f0c26" (UID: "3bea2f06-3c8b-4d55-aeea-91c8820f0c26"). InnerVolumeSpecName "kube-api-access-ssq2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:00:03.820773 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.820737 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59j94\" (UniqueName: \"kubernetes.io/projected/c9e53efa-bf16-49d7-a1e0-78d5acfba367-kube-api-access-59j94\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:00:03.820773 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.820771 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssq2x\" (UniqueName: \"kubernetes.io/projected/3bea2f06-3c8b-4d55-aeea-91c8820f0c26-kube-api-access-ssq2x\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:00:03.820773 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:03.820780 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776kr\" (UniqueName: \"kubernetes.io/projected/0c42829b-9efc-43b1-bbc8-21617c31596d-kube-api-access-776kr\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:00:04.410564 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.410536 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" Jul 12 20:00:04.410564 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.410548 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr" event={"ID":"0c42829b-9efc-43b1-bbc8-21617c31596d","Type":"ContainerDied","Data":"063fb1c92ceab89c5bcc55d473627403825d70bebce3f06f7772a70434c326e6"} Jul 12 20:00:04.410776 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.410599 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="063fb1c92ceab89c5bcc55d473627403825d70bebce3f06f7772a70434c326e6" Jul 12 20:00:04.411852 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.411833 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" Jul 12 20:00:04.411978 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.411878 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29731440-bjbtm" event={"ID":"c9e53efa-bf16-49d7-a1e0-78d5acfba367","Type":"ContainerDied","Data":"822526ce9b79abd9c607ddbc56e91900ca5a76170211a83bd0837a43044bff91"} Jul 12 20:00:04.411978 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.411925 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="822526ce9b79abd9c607ddbc56e91900ca5a76170211a83bd0837a43044bff91" Jul 12 20:00:04.413366 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.413344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" event={"ID":"3bea2f06-3c8b-4d55-aeea-91c8820f0c26","Type":"ContainerDied","Data":"40e6aefa1e9383d0571046185121f98491e7a5a3517a928d7d3292b465d704fb"} Jul 12 20:00:04.413463 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.413378 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40e6aefa1e9383d0571046185121f98491e7a5a3517a928d7d3292b465d704fb" Jul 12 20:00:04.413463 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:04.413349 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767" Jul 12 20:00:05.769395 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:05.769361 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:00:21.975664 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:21.975625 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:00:33.972489 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:33.972443 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:00:52.031751 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.031663 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl"] Jul 12 20:00:52.032622 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032598 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c42829b-9efc-43b1-bbc8-21617c31596d" containerName="cleanup" Jul 12 20:00:52.032622 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032622 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c42829b-9efc-43b1-bbc8-21617c31596d" containerName="cleanup" Jul 12 20:00:52.032802 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032635 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bea2f06-3c8b-4d55-aeea-91c8820f0c26" containerName="cleanup" Jul 12 20:00:52.032802 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032644 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bea2f06-3c8b-4d55-aeea-91c8820f0c26" containerName="cleanup" Jul 12 20:00:52.032802 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032681 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9e53efa-bf16-49d7-a1e0-78d5acfba367" containerName="cleanup" Jul 12 20:00:52.032802 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032689 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9e53efa-bf16-49d7-a1e0-78d5acfba367" containerName="cleanup" Jul 12 20:00:52.032802 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032788 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9e53efa-bf16-49d7-a1e0-78d5acfba367" containerName="cleanup" Jul 12 20:00:52.032802 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032800 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c42829b-9efc-43b1-bbc8-21617c31596d" containerName="cleanup" Jul 12 20:00:52.033119 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.032809 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bea2f06-3c8b-4d55-aeea-91c8820f0c26" containerName="cleanup" Jul 12 20:00:52.036343 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.036324 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.039827 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.039804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"sub-test-model-5865e5-kserve-self-signed-certs\"" Jul 12 20:00:52.039952 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.039804 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"kube-root-ca.crt\"" Jul 12 20:00:52.039952 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.039804 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"default-dockercfg-t2pbn\"" Jul 12 20:00:52.039952 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.039904 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-5865e5\"/\"openshift-service-ca.crt\"" Jul 12 20:00:52.043725 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.043706 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl"] Jul 12 20:00:52.141909 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.141867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-dshm\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.141909 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.141908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tls-certs\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.142123 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.141929 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prnbq\" (UniqueName: \"kubernetes.io/projected/ad77a5a6-9a14-414a-a89a-7b0922d404ff-kube-api-access-prnbq\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.142123 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.142015 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-home\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.142123 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.142077 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-model-cache\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.142123 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.142104 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tmp-dir\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.242872 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.242837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-home\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243079 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.242906 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-model-cache\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243079 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.242928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tmp-dir\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243079 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.242992 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-dshm\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243079 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.243014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tls-certs\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243079 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.243040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prnbq\" (UniqueName: \"kubernetes.io/projected/ad77a5a6-9a14-414a-a89a-7b0922d404ff-kube-api-access-prnbq\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243365 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.243334 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-home\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243431 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.243368 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-model-cache\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.243471 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.243431 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tmp-dir\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.245338 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.245318 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-dshm\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.245484 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.245468 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tls-certs\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.250617 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.250594 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prnbq\" (UniqueName: \"kubernetes.io/projected/ad77a5a6-9a14-414a-a89a-7b0922d404ff-kube-api-access-prnbq\") pod \"sub-test-model-5865e5-kserve-55946bcbc-j2bwl\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.347728 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.347699 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:00:52.489241 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.489217 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl"] Jul 12 20:00:52.491371 ip-10-0-141-249 kubenswrapper[2565]: W0712 20:00:52.491344 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad77a5a6_9a14_414a_a89a_7b0922d404ff.slice/crio-0d8373b3523cdb3a87c3f20c826b8d54b53c1278d50635e2f76f573b473aca13 WatchSource:0}: Error finding container 0d8373b3523cdb3a87c3f20c826b8d54b53c1278d50635e2f76f573b473aca13: Status 404 returned error can't find the container with id 0d8373b3523cdb3a87c3f20c826b8d54b53c1278d50635e2f76f573b473aca13 Jul 12 20:00:52.666933 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.666835 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" event={"ID":"ad77a5a6-9a14-414a-a89a-7b0922d404ff","Type":"ContainerStarted","Data":"b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1"} Jul 12 20:00:52.666933 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.666880 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" event={"ID":"ad77a5a6-9a14-414a-a89a-7b0922d404ff","Type":"ContainerStarted","Data":"0d8373b3523cdb3a87c3f20c826b8d54b53c1278d50635e2f76f573b473aca13"} Jul 12 20:00:52.682453 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:52.682393 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" podStartSLOduration=0.682375555 podStartE2EDuration="682.375555ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-12 20:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:00:52.681453755 +0000 UTC m=+2341.363932216" watchObservedRunningTime="2026-07-12 20:00:52.682375555 +0000 UTC m=+2341.364854014" Jul 12 20:00:54.975034 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:00:54.974991 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:01:02.348542 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:02.348502 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:01:02.348962 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:02.348557 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:01:02.361719 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:02.361688 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:01:02.727400 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:02.727315 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:01:08.973270 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:08.973231 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:01:25.145673 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:25.145635 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl"] Jul 12 20:01:28.874741 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:28.874708 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:01:40.167069 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.167026 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" podUID="ad77a5a6-9a14-414a-a89a-7b0922d404ff" containerName="main" containerID="cri-o://b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1" gracePeriod=45 Jul 12 20:01:40.434020 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.433993 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:01:40.570246 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570215 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-model-cache\") pod \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " Jul 12 20:01:40.570450 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570254 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prnbq\" (UniqueName: \"kubernetes.io/projected/ad77a5a6-9a14-414a-a89a-7b0922d404ff-kube-api-access-prnbq\") pod \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " Jul 12 20:01:40.570450 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570290 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-dshm\") pod \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " Jul 12 20:01:40.570450 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570346 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tls-certs\") pod \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " Jul 12 20:01:40.570450 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570382 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-home\") pod \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " Jul 12 20:01:40.570450 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570416 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tmp-dir\") pod \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\" (UID: \"ad77a5a6-9a14-414a-a89a-7b0922d404ff\") " Jul 12 20:01:40.570715 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570521 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad77a5a6-9a14-414a-a89a-7b0922d404ff" (UID: "ad77a5a6-9a14-414a-a89a-7b0922d404ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:40.570715 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570703 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad77a5a6-9a14-414a-a89a-7b0922d404ff" (UID: "ad77a5a6-9a14-414a-a89a-7b0922d404ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:40.570800 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570720 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-model-cache\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:01:40.570800 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.570717 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-home" (OuterVolumeSpecName: "home") pod "ad77a5a6-9a14-414a-a89a-7b0922d404ff" (UID: "ad77a5a6-9a14-414a-a89a-7b0922d404ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:40.572586 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.572563 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-dshm" (OuterVolumeSpecName: "dshm") pod "ad77a5a6-9a14-414a-a89a-7b0922d404ff" (UID: "ad77a5a6-9a14-414a-a89a-7b0922d404ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 12 20:01:40.572669 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.572609 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad77a5a6-9a14-414a-a89a-7b0922d404ff-kube-api-access-prnbq" (OuterVolumeSpecName: "kube-api-access-prnbq") pod "ad77a5a6-9a14-414a-a89a-7b0922d404ff" (UID: "ad77a5a6-9a14-414a-a89a-7b0922d404ff"). InnerVolumeSpecName "kube-api-access-prnbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:01:40.572711 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.572672 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad77a5a6-9a14-414a-a89a-7b0922d404ff" (UID: "ad77a5a6-9a14-414a-a89a-7b0922d404ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:01:40.671637 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.671554 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tls-certs\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:01:40.671637 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.671582 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-home\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:01:40.671637 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.671593 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-tmp-dir\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:01:40.671637 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.671602 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prnbq\" (UniqueName: \"kubernetes.io/projected/ad77a5a6-9a14-414a-a89a-7b0922d404ff-kube-api-access-prnbq\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:01:40.671637 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.671612 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad77a5a6-9a14-414a-a89a-7b0922d404ff-dshm\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:01:40.908129 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.908091 2565 generic.go:356] "Generic (PLEG): container finished" podID="ad77a5a6-9a14-414a-a89a-7b0922d404ff" containerID="b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1" exitCode=0 Jul 12 20:01:40.908334 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.908135 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" Jul 12 20:01:40.908334 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.908144 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" event={"ID":"ad77a5a6-9a14-414a-a89a-7b0922d404ff","Type":"ContainerDied","Data":"b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1"} Jul 12 20:01:40.908334 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.908208 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl" event={"ID":"ad77a5a6-9a14-414a-a89a-7b0922d404ff","Type":"ContainerDied","Data":"0d8373b3523cdb3a87c3f20c826b8d54b53c1278d50635e2f76f573b473aca13"} Jul 12 20:01:40.908334 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.908232 2565 scope.go:117] "RemoveContainer" containerID="b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1" Jul 12 20:01:40.917799 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.917781 2565 scope.go:117] "RemoveContainer" containerID="b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1" Jul 12 20:01:40.918049 ip-10-0-141-249 kubenswrapper[2565]: E0712 20:01:40.918031 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1\": container with ID starting with b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1 not found: ID does not exist" containerID="b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1" Jul 12 20:01:40.918098 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.918056 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1"} err="failed to get container status \"b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1\": rpc error: code = NotFound desc = could not find container \"b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1\": container with ID starting with b9a236f0999438f94bb4d5decb66c036b300614cacf0ede460a7e92a3ad4fcf1 not found: ID does not exist" Jul 12 20:01:40.928497 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.928441 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl"] Jul 12 20:01:40.931826 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:40.931806 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-5865e5/sub-test-model-5865e5-kserve-55946bcbc-j2bwl"] Jul 12 20:01:41.862117 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:41.862082 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad77a5a6-9a14-414a-a89a-7b0922d404ff" path="/var/lib/kubelet/pods/ad77a5a6-9a14-414a-a89a-7b0922d404ff/volumes" Jul 12 20:01:43.476480 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:43.476442 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:01:51.951531 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:51.951499 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 20:01:51.954019 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:51.953991 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 20:01:51.955849 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:51.955828 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 20:01:51.958579 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:01:51.958560 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 20:02:07.575498 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:02:07.575464 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:02:30.277070 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:02:30.277038 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:02:43.670989 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:02:43.670955 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:03:00.076820 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:00.076783 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:03:58.714048 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.713954 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74"] Jul 12 20:03:58.714630 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.714211 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" podUID="9909c9bd-3cb6-4896-b1fb-3e657df3eee5" containerName="maas-api" containerID="cri-o://a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361" gracePeriod=30 Jul 12 20:03:58.737089 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.737062 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr"] Jul 12 20:03:58.739431 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.739407 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-a-e68438-29731440-xgfpr"] Jul 12 20:03:58.971789 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.971728 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 20:03:58.991242 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.991213 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-openshift-service-ca\") pod \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " Jul 12 20:03:58.991426 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.991268 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls\") pod \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " Jul 12 20:03:58.991426 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.991299 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddnwj\" (UniqueName: \"kubernetes.io/projected/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-kube-api-access-ddnwj\") pod \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\" (UID: \"9909c9bd-3cb6-4896-b1fb-3e657df3eee5\") " Jul 12 20:03:58.991631 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.991609 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9909c9bd-3cb6-4896-b1fb-3e657df3eee5" (UID: "9909c9bd-3cb6-4896-b1fb-3e657df3eee5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 20:03:58.993432 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.993406 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-kube-api-access-ddnwj" (OuterVolumeSpecName: "kube-api-access-ddnwj") pod "9909c9bd-3cb6-4896-b1fb-3e657df3eee5" (UID: "9909c9bd-3cb6-4896-b1fb-3e657df3eee5"). InnerVolumeSpecName "kube-api-access-ddnwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:03:58.993556 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:58.993498 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9909c9bd-3cb6-4896-b1fb-3e657df3eee5" (UID: "9909c9bd-3cb6-4896-b1fb-3e657df3eee5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:03:59.092232 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.092172 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddnwj\" (UniqueName: \"kubernetes.io/projected/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-kube-api-access-ddnwj\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:03:59.092232 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.092226 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:03:59.092232 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.092236 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9909c9bd-3cb6-4896-b1fb-3e657df3eee5-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:03:59.591763 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.591726 2565 generic.go:356] "Generic (PLEG): container finished" podID="9909c9bd-3cb6-4896-b1fb-3e657df3eee5" containerID="a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361" exitCode=0 Jul 12 20:03:59.591964 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.591779 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" Jul 12 20:03:59.591964 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.591809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" event={"ID":"9909c9bd-3cb6-4896-b1fb-3e657df3eee5","Type":"ContainerDied","Data":"a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361"} Jul 12 20:03:59.591964 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.591895 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74" event={"ID":"9909c9bd-3cb6-4896-b1fb-3e657df3eee5","Type":"ContainerDied","Data":"7d79a8852698377a5c7fdc028a52dcfc7cc67ba79b35270916d5c9fec9e9bb44"} Jul 12 20:03:59.591964 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.591928 2565 scope.go:117] "RemoveContainer" containerID="a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361" Jul 12 20:03:59.601149 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.601133 2565 scope.go:117] "RemoveContainer" containerID="a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361" Jul 12 20:03:59.601432 ip-10-0-141-249 kubenswrapper[2565]: E0712 20:03:59.601413 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361\": container with ID starting with a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361 not found: ID does not exist" containerID="a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361" Jul 12 20:03:59.601500 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.601439 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361"} err="failed to get container status \"a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361\": rpc error: code = NotFound desc = could not find container \"a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361\": container with ID starting with a6b2a8ee03560c8fc107e616b6b2cd4dab0073e3c067cab123e7e76c66abe361 not found: ID does not exist" Jul 12 20:03:59.612134 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.612102 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74"] Jul 12 20:03:59.615865 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.615843 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-a-e68438-d5bdfff9d-f8r74"] Jul 12 20:03:59.861901 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.861818 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c42829b-9efc-43b1-bbc8-21617c31596d" path="/var/lib/kubelet/pods/0c42829b-9efc-43b1-bbc8-21617c31596d/volumes" Jul 12 20:03:59.862277 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:03:59.862219 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9909c9bd-3cb6-4896-b1fb-3e657df3eee5" path="/var/lib/kubelet/pods/9909c9bd-3cb6-4896-b1fb-3e657df3eee5/volumes" Jul 12 20:04:02.275416 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:02.275380 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:04:05.462061 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.462026 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx"] Jul 12 20:04:05.462557 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.462287 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" podUID="8bd8a707-00d1-4c76-a37d-590c37d37ac4" containerName="maas-api" containerID="cri-o://2a30575504fb243a0a0fbe5b72a5b4615c2085f752de821631281d7ae9c7ba7d" gracePeriod=30 Jul 12 20:04:05.488787 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.488696 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767"] Jul 12 20:04:05.494266 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.492140 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-shared-b-5865e5-29731440-5w767"] Jul 12 20:04:05.624385 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.624338 2565 generic.go:356] "Generic (PLEG): container finished" podID="8bd8a707-00d1-4c76-a37d-590c37d37ac4" containerID="2a30575504fb243a0a0fbe5b72a5b4615c2085f752de821631281d7ae9c7ba7d" exitCode=0 Jul 12 20:04:05.624534 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.624410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" event={"ID":"8bd8a707-00d1-4c76-a37d-590c37d37ac4","Type":"ContainerDied","Data":"2a30575504fb243a0a0fbe5b72a5b4615c2085f752de821631281d7ae9c7ba7d"} Jul 12 20:04:05.720101 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.720035 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 20:04:05.749627 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.749597 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd8a707-00d1-4c76-a37d-590c37d37ac4-openshift-service-ca\") pod \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " Jul 12 20:04:05.749791 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.749674 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls\") pod \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " Jul 12 20:04:05.749880 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.749834 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbtvh\" (UniqueName: \"kubernetes.io/projected/8bd8a707-00d1-4c76-a37d-590c37d37ac4-kube-api-access-mbtvh\") pod \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\" (UID: \"8bd8a707-00d1-4c76-a37d-590c37d37ac4\") " Jul 12 20:04:05.749969 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.749945 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd8a707-00d1-4c76-a37d-590c37d37ac4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bd8a707-00d1-4c76-a37d-590c37d37ac4" (UID: "8bd8a707-00d1-4c76-a37d-590c37d37ac4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 12 20:04:05.750198 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.750157 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bd8a707-00d1-4c76-a37d-590c37d37ac4-openshift-service-ca\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:04:05.751731 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.751710 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bd8a707-00d1-4c76-a37d-590c37d37ac4" (UID: "8bd8a707-00d1-4c76-a37d-590c37d37ac4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 12 20:04:05.751961 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.751936 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd8a707-00d1-4c76-a37d-590c37d37ac4-kube-api-access-mbtvh" (OuterVolumeSpecName: "kube-api-access-mbtvh") pod "8bd8a707-00d1-4c76-a37d-590c37d37ac4" (UID: "8bd8a707-00d1-4c76-a37d-590c37d37ac4"). InnerVolumeSpecName "kube-api-access-mbtvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 12 20:04:05.850864 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.850817 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bd8a707-00d1-4c76-a37d-590c37d37ac4-maas-api-tls\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:04:05.850864 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.850850 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbtvh\" (UniqueName: \"kubernetes.io/projected/8bd8a707-00d1-4c76-a37d-590c37d37ac4-kube-api-access-mbtvh\") on node \"ip-10-0-141-249.ec2.internal\" DevicePath \"\"" Jul 12 20:04:05.863199 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:05.863147 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bea2f06-3c8b-4d55-aeea-91c8820f0c26" path="/var/lib/kubelet/pods/3bea2f06-3c8b-4d55-aeea-91c8820f0c26/volumes" Jul 12 20:04:06.630497 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:06.630472 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" Jul 12 20:04:06.630497 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:06.630490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx" event={"ID":"8bd8a707-00d1-4c76-a37d-590c37d37ac4","Type":"ContainerDied","Data":"05f970a0be5c80527c424c9528813714b75c17cc71268677386c60149bebfe88"} Jul 12 20:04:06.630997 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:06.630543 2565 scope.go:117] "RemoveContainer" containerID="2a30575504fb243a0a0fbe5b72a5b4615c2085f752de821631281d7ae9c7ba7d" Jul 12 20:04:06.647489 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:06.647466 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx"] Jul 12 20:04:06.650792 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:06.650772 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-5865e5-6665ff885-fqpsx"] Jul 12 20:04:07.862573 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:07.862541 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bd8a707-00d1-4c76-a37d-590c37d37ac4" path="/var/lib/kubelet/pods/8bd8a707-00d1-4c76-a37d-590c37d37ac4/volumes" Jul 12 20:04:14.772122 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:14.772080 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-qcz8n"] Jul 12 20:04:16.454663 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:16.454629 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t6r72_fb227f05-3565-4d27-b7f0-25a0a1ed13cd/manager/0.log" Jul 12 20:04:16.567783 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:16.567751 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-867fr_53d57987-29a7-468b-8d48-fa747749f992/manager/0.log" Jul 12 20:04:16.681246 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:16.681212 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-57bdd766bb-cxs9w_c84c2eb5-2a76-432c-8ffd-df78e511e186/maas-api/0.log" Jul 12 20:04:16.784511 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:16.784426 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29731425-bmhlc_a7386590-582f-48b4-bdd7-769be41b6815/cleanup/0.log" Jul 12 20:04:16.888073 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:16.888046 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29731440-bjbtm_c9e53efa-bf16-49d7-a1e0-78d5acfba367/cleanup/0.log" Jul 12 20:04:17.000964 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:17.000932 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-557cf6647c-rz92f_7142d987-c46d-4bdc-92b9-72cd1c35f76a/manager/0.log" Jul 12 20:04:17.559888 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:17.559857 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-zvhfd_018d09cc-2ee0-436f-841c-cb4506fe05d1/copy-manifests/0.log" Jul 12 20:04:17.572490 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:17.572463 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-zvhfd_018d09cc-2ee0-436f-841c-cb4506fe05d1/manager/0.log" Jul 12 20:04:19.608386 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:19.608350 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-qcz8n_ed027eec-553f-4c48-b37d-023b29c4be1c/limitador/0.log" Jul 12 20:04:20.841233 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:20.841202 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86d8bc5964-wgq5f_928c8c62-9201-42ec-b710-560d624ec55c/router/0.log" Jul 12 20:04:21.163973 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:21.163884 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-bmjdp_62845992-558f-4686-96f2-5c26da67a74a/main/0.log" Jul 12 20:04:21.271661 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:21.271634 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5rc9q_d7a91b11-9003-4d1d-9833-73026fc788b5/main/0.log" Jul 12 20:04:29.069350 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:29.069318 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bblrg_c57f6e17-8e69-4af8-9a1e-540b29c4874c/global-pull-secret-syncer/0.log" Jul 12 20:04:29.188975 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:29.188946 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dqlj7_9575ef7a-efbb-434d-a849-60c5d68d5f0f/konnectivity-agent/0.log" Jul 12 20:04:29.252782 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:29.252754 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-249.ec2.internal_8c0de6f6d239334ca28d971fb295f674/haproxy/0.log" Jul 12 20:04:34.746644 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:34.746616 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-qcz8n_ed027eec-553f-4c48-b37d-023b29c4be1c/limitador/0.log" Jul 12 20:04:36.681157 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:36.681125 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/1.log" Jul 12 20:04:36.762005 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:36.761980 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-85cdb_df8f3888-df8e-480d-ae68-f6b16126ee83/cluster-monitoring-operator/0.log" Jul 12 20:04:37.080164 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:37.080090 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd8c7_b0b133e1-6e87-423e-a535-745aa9beaeca/node-exporter/0.log" Jul 12 20:04:37.102335 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:37.102312 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd8c7_b0b133e1-6e87-423e-a535-745aa9beaeca/kube-rbac-proxy/0.log" Jul 12 20:04:37.153747 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:37.153721 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sd8c7_b0b133e1-6e87-423e-a535-745aa9beaeca/init-textfile/0.log" Jul 12 20:04:38.856292 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:38.856253 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-5cb5t_7d336bda-ff92-4cd6-be90-8dba32805f55/networking-console-plugin/0.log" Jul 12 20:04:39.119444 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.119362 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk"] Jul 12 20:04:39.119967 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.119816 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9909c9bd-3cb6-4896-b1fb-3e657df3eee5" containerName="maas-api" Jul 12 20:04:39.119967 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.119841 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9909c9bd-3cb6-4896-b1fb-3e657df3eee5" containerName="maas-api" Jul 12 20:04:39.119967 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.119855 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bd8a707-00d1-4c76-a37d-590c37d37ac4" containerName="maas-api" Jul 12 20:04:39.119967 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.119862 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bd8a707-00d1-4c76-a37d-590c37d37ac4" containerName="maas-api" Jul 12 20:04:39.119967 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.119878 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad77a5a6-9a14-414a-a89a-7b0922d404ff" containerName="main" Jul 12 20:04:39.119967 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.119886 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad77a5a6-9a14-414a-a89a-7b0922d404ff" containerName="main" Jul 12 20:04:39.120282 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.120002 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9909c9bd-3cb6-4896-b1fb-3e657df3eee5" containerName="maas-api" Jul 12 20:04:39.120282 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.120012 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad77a5a6-9a14-414a-a89a-7b0922d404ff" containerName="main" Jul 12 20:04:39.120282 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.120019 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bd8a707-00d1-4c76-a37d-590c37d37ac4" containerName="maas-api" Jul 12 20:04:39.123425 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.123406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.125752 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.125733 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wqrhv\"/\"default-dockercfg-jbd6q\"" Jul 12 20:04:39.126714 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.126696 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wqrhv\"/\"kube-root-ca.crt\"" Jul 12 20:04:39.126851 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.126767 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wqrhv\"/\"openshift-service-ca.crt\"" Jul 12 20:04:39.131875 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.131854 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk"] Jul 12 20:04:39.238472 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.238434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-proc\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.238472 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.238468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-podres\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.238687 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.238489 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-sys\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.238687 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.238588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-lib-modules\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.238687 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.238624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhxb\" (UniqueName: \"kubernetes.io/projected/fab80b7c-4aa5-43fc-8d0b-2b4574473072-kube-api-access-sbhxb\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.339783 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-proc\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.339783 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339782 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-podres\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.339783 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339802 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-sys\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.340068 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339845 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-lib-modules\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.340068 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhxb\" (UniqueName: \"kubernetes.io/projected/fab80b7c-4aa5-43fc-8d0b-2b4574473072-kube-api-access-sbhxb\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.340068 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339878 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-proc\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.340068 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339916 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-podres\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.340068 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.339944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-sys\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.340068 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.340034 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fab80b7c-4aa5-43fc-8d0b-2b4574473072-lib-modules\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.346681 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.346658 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhxb\" (UniqueName: \"kubernetes.io/projected/fab80b7c-4aa5-43fc-8d0b-2b4574473072-kube-api-access-sbhxb\") pod \"perf-node-gather-daemonset-xxnmk\" (UID: \"fab80b7c-4aa5-43fc-8d0b-2b4574473072\") " pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.435134 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.435025 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.466774 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.466724 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/1.log" Jul 12 20:04:39.472324 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.472298 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-lcxrh_c3551a07-e183-40a5-9b98-1a6865047fb9/console-operator/2.log" Jul 12 20:04:39.568421 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.568397 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk"] Jul 12 20:04:39.571432 ip-10-0-141-249 kubenswrapper[2565]: W0712 20:04:39.571408 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfab80b7c_4aa5_43fc_8d0b_2b4574473072.slice/crio-368c8a4dcc5417f3ab740be3e90c81725d64e85d33df12dfb8fb09e4cce1fca7 WatchSource:0}: Error finding container 368c8a4dcc5417f3ab740be3e90c81725d64e85d33df12dfb8fb09e4cce1fca7: Status 404 returned error can't find the container with id 368c8a4dcc5417f3ab740be3e90c81725d64e85d33df12dfb8fb09e4cce1fca7 Jul 12 20:04:39.796138 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.796099 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" event={"ID":"fab80b7c-4aa5-43fc-8d0b-2b4574473072","Type":"ContainerStarted","Data":"e71dba6e44dd69508e2bf61b910d8cae76387fc4dfa1a853fb57e980414157f0"} Jul 12 20:04:39.796346 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.796160 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" event={"ID":"fab80b7c-4aa5-43fc-8d0b-2b4574473072","Type":"ContainerStarted","Data":"368c8a4dcc5417f3ab740be3e90c81725d64e85d33df12dfb8fb09e4cce1fca7"} Jul 12 20:04:39.796346 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.796212 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:39.811794 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:39.811742 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" podStartSLOduration=0.811726168 podStartE2EDuration="811.726168ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-12 20:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-12 20:04:39.80974147 +0000 UTC m=+2568.492219930" watchObservedRunningTime="2026-07-12 20:04:39.811726168 +0000 UTC m=+2568.494204627" Jul 12 20:04:40.044530 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:40.044449 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-f6ng5_5e13f450-f8cd-43aa-9864-9670f84317ec/download-server/0.log" Jul 12 20:04:40.612584 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:40.612560 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-vd82r_894b53d3-e0f9-4c31-b48f-8d2f80edd455/volume-data-source-validator/0.log" Jul 12 20:04:41.596817 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:41.596787 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmpsl_566fd99f-bbc7-4532-83ef-f8cce366f737/dns/0.log" Jul 12 20:04:41.617193 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:41.617152 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lmpsl_566fd99f-bbc7-4532-83ef-f8cce366f737/kube-rbac-proxy/0.log" Jul 12 20:04:41.720023 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:41.720000 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n6xsf_320c816e-c163-4ccb-a5fd-a6e8d2eb88df/dns-node-resolver/0.log" Jul 12 20:04:42.341162 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:42.341130 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zgx4p_43101f96-47a6-4a66-b3bc-36244a83785b/node-ca/0.log" Jul 12 20:04:43.628586 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:43.628559 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86d8bc5964-wgq5f_928c8c62-9201-42ec-b710-560d624ec55c/router/0.log" Jul 12 20:04:44.272002 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:44.271975 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6pvxk_91f6dd18-06c6-4622-890d-3865aae784de/serve-healthcheck-canary/0.log" Jul 12 20:04:44.747462 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:44.747432 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-52s44_6aa31e23-7983-4949-b4cc-4fe996a91a92/insights-operator/0.log" Jul 12 20:04:44.748098 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:44.748082 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-52s44_6aa31e23-7983-4949-b4cc-4fe996a91a92/insights-operator/1.log" Jul 12 20:04:44.768442 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:44.768419 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dm4ws_53fba168-256f-4c00-a11a-69b3a3224fff/kube-rbac-proxy/0.log" Jul 12 20:04:44.789026 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:44.788996 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dm4ws_53fba168-256f-4c00-a11a-69b3a3224fff/exporter/0.log" Jul 12 20:04:44.808719 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:44.808693 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dm4ws_53fba168-256f-4c00-a11a-69b3a3224fff/extractor/0.log" Jul 12 20:04:45.811410 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:45.811382 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wqrhv/perf-node-gather-daemonset-xxnmk" Jul 12 20:04:46.995859 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:46.995824 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t6r72_fb227f05-3565-4d27-b7f0-25a0a1ed13cd/manager/0.log" Jul 12 20:04:47.116325 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:47.116295 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-867fr_53d57987-29a7-468b-8d48-fa747749f992/manager/0.log" Jul 12 20:04:47.150937 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:47.150904 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-57bdd766bb-cxs9w_c84c2eb5-2a76-432c-8ffd-df78e511e186/maas-api/0.log" Jul 12 20:04:47.169379 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:47.169346 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29731425-bmhlc_a7386590-582f-48b4-bdd7-769be41b6815/cleanup/0.log" Jul 12 20:04:47.191615 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:47.191587 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29731440-bjbtm_c9e53efa-bf16-49d7-a1e0-78d5acfba367/cleanup/0.log" Jul 12 20:04:47.290569 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:47.290482 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-557cf6647c-rz92f_7142d987-c46d-4bdc-92b9-72cd1c35f76a/manager/0.log" Jul 12 20:04:47.577452 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:47.577370 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-zvhfd_018d09cc-2ee0-436f-841c-cb4506fe05d1/manager/0.log" Jul 12 20:04:47.593962 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:47.593935 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5449d7969f-zvhfd_018d09cc-2ee0-436f-841c-cb4506fe05d1/copy-manifests/0.log" Jul 12 20:04:49.352138 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:49.352112 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-79f8975998-rj5jw_e42822c4-13e0-4723-a43f-1e9be58e4e4b/manager/0.log" Jul 12 20:04:50.428554 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:50.428471 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86d8bc5964-wgq5f_928c8c62-9201-42ec-b710-560d624ec55c/router/0.log" Jul 12 20:04:51.924867 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:51.924837 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86d8bc5964-wgq5f_928c8c62-9201-42ec-b710-560d624ec55c/router/0.log" Jul 12 20:04:54.310650 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:54.310619 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-p2q5j_83f1366d-c723-4434-a108-1093df72c064/migrator/0.log" Jul 12 20:04:54.330477 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:54.330446 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-p2q5j_83f1366d-c723-4434-a108-1093df72c064/graceful-termination/0.log" Jul 12 20:04:54.680221 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:54.680171 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-vjfb7_d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac/kube-storage-version-migrator-operator/1.log" Jul 12 20:04:54.681564 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:54.681530 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-vjfb7_d5f6fff9-f604-4780-bbf0-4a1f6e6fc9ac/kube-storage-version-migrator-operator/0.log" Jul 12 20:04:55.635001 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.634964 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-79cw7_0107542d-e0e6-4cbe-bc59-0ff3ed1ac8b7/kube-multus/0.log" Jul 12 20:04:55.823606 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.823578 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwllz_9710beb1-c58c-4188-92b1-61ebc42865e2/kube-multus-additional-cni-plugins/0.log" Jul 12 20:04:55.845240 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.845213 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwllz_9710beb1-c58c-4188-92b1-61ebc42865e2/egress-router-binary-copy/0.log" Jul 12 20:04:55.864922 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.864898 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwllz_9710beb1-c58c-4188-92b1-61ebc42865e2/cni-plugins/0.log" Jul 12 20:04:55.883894 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.883868 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwllz_9710beb1-c58c-4188-92b1-61ebc42865e2/bond-cni-plugin/0.log" Jul 12 20:04:55.903250 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.903190 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwllz_9710beb1-c58c-4188-92b1-61ebc42865e2/routeoverride-cni/0.log" Jul 12 20:04:55.923029 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.923004 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwllz_9710beb1-c58c-4188-92b1-61ebc42865e2/whereabouts-cni-bincopy/0.log" Jul 12 20:04:55.941478 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:55.941453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dwllz_9710beb1-c58c-4188-92b1-61ebc42865e2/whereabouts-cni/0.log" Jul 12 20:04:56.144716 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:56.144684 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ggg8r_a8899441-0b5a-4714-9743-f03b4cc1ed86/network-metrics-daemon/0.log" Jul 12 20:04:56.164067 ip-10-0-141-249 kubenswrapper[2565]: I0712 20:04:56.163994 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ggg8r_a8899441-0b5a-4714-9743-f03b4cc1ed86/kube-rbac-proxy/0.log"